High-throughput computing: Difference between revisions
| (33 intermediate revisions by 25 users not shown) | |||
| Line 1: | Line 1: | ||
| ⚫ | |||
{{Short description|Use of many computing resources over long periods of time to accomplish a computational task}} |
|||
| ⚫ | |||
==Challenges== |
==Challenges== |
||
The HTC community is also concerned with robustness and reliability of jobs over a long-time scale. That is, being able to create a reliable system from unreliable components. This research is similar to [[transaction processing]], but at a much larger and distributed scale. |
The HTC community is also concerned with robustness and reliability of jobs over a long-time scale. That is, being able to create a reliable system from unreliable components. This research is similar to [[transaction processing]], but at a much larger and distributed scale. |
||
Some HTC systems, such as [[ |
Some HTC systems, such as [[HTCondor]] and [[Portable Batch System|PBS]], can run tasks on opportunistic resources. It is a difficult problem, however, to operate in this environment. On one hand the system needs to provide a reliable operating environment for the user's jobs, but at the same time the system must not compromise the integrity of the execute node and allow the owner to always have full control of their resources. |
||
==High-throughput |
==High-throughput vs. high-performance vs. many-task== |
||
There are many differences between high-throughput computing, [[high-performance computing]] (HPC), and [[many-task computing]] (MTC). |
|||
There are many differences between '''high-throughput computing''', [[high-performance computing]] (HPC), and [[many-task computing]] (MTC). HPC tasks are characterized as needing large of amounts of computing power for short periods of time, whereas HTC tasks also require large amounts of computing, but for much longer times (months and years, rather than hours and days) <ref>{{cite web|url=http://www.hpcwire.com/hpc-bin/artread.pl?direction=Current&articlenumber=11444|first=Alan|last=Beck|date = [[1997-06-27]]|publisher=HPCWire|title=High Throughput Computing: An Interview with Miron Livny}}</ref>. HPC environments are often measured in terms of [[FLOPS]]. The HTC community, however, is not concerned about operations per second, but rather operations per month or per year. Therefore, the HTC field is more interested in how many jobs can be completed over a long period of time instead of how fast an individual job can complete. |
|||
HPC tasks are characterized as needing large amounts of computing power for short periods of time, whereas HTC tasks also require large amounts of computing, but for much longer times (months and years, rather than hours and days).<ref>{{cite web|url=https://www.hpcwire.com/1997/06/27/high-throughput-computing-an-interview-with-miron-livny/|first=Alan|last=Beck|date = 1997-06-27|publisher=HPCWire|title=High Throughput Computing: An Interview with Miron Livny (Broken link)}}</ref><ref>{{cite web|url=http://research.cs.wisc.edu/condor/HPCwire.1|title=High Throughput Computing: An Interview with Miron Livny (Backup link)}}</ref> HPC environments are often measured in terms of [[FLOPS]]. |
|||
| ⚫ | As a |
||
The HTC community, however, is not concerned about operations per second, but rather operations per month or per year. Therefore, the HTC field is more interested in how many jobs can be completed over a long period of time instead of how fast. |
|||
| ⚫ | As an alternative definition, the [[European Grid Infrastructure]] defines HTC as ''"a computing paradigm that focuses on the efficient execution of a large number of loosely-coupled tasks"'',<ref>{{cite web|url=https://wiki.egi.eu/wiki/Glossary_V1#High_Throughput_Computing|title=EGI Glossary V1}}</ref> while HPC systems tend to focus on tightly coupled [[parallel computing|parallel]] jobs, and as such they must execute within a particular site with low-latency interconnects. Conversely, HTC systems are independent, sequential jobs that can be individually scheduled on many different computing resources across multiple administrative boundaries. HTC systems achieve this using various [[grid computing]] technologies and techniques. |
||
MTC aims to bridge the gap between HTC and HPC. MTC is reminiscent of HTC, but it differs in the emphasis of using many computing resources over short periods of time to accomplish many computational tasks (i.e. including both dependent and independent tasks), where the primary metrics are measured in seconds (e.g. FLOPS, tasks/s, MB/s I/O rates), as opposed to operations (e.g. jobs) per month. MTC denotes high-performance computations comprising multiple distinct activities, coupled via file system operations. |
MTC aims to bridge the gap between HTC and HPC. MTC is reminiscent of HTC, but it differs in the emphasis of using many computing resources over short periods of time to accomplish many computational tasks (i.e. including both dependent and independent tasks), where the primary metrics are measured in seconds (e.g. FLOPS, tasks/s, MB/s I/O rates), as opposed to operations (e.g. jobs) per month. MTC denotes high-performance computations comprising multiple distinct activities, coupled via file system operations. |
||
| Line 15: | Line 22: | ||
==See also== |
==See also== |
||
* [[Batch processing]] |
* [[Batch processing]] |
||
* [[e-Science]] |
|||
* [[Condor High-Throughput Computing System]] |
|||
* [[Open Science Grid Consortium]] |
|||
HTC is not only for sequential jobs but also for parallel jobs. |
|||
* [[Grid computing]] |
|||
==References== |
==References== |
||
{{Reflist}} |
|||
<references/> |
|||
==External links== |
|||
*[http://www.cs.wisc.edu/condor/htc.html Condor's Definition of high throughput computing] |
|||
[[Category:Parallel computing]] |
[[Category:Parallel computing]] |
||
Latest revision as of 13:22, 19 July 2023
In computer science, high-throughput computing (HTC) is the use of many computing resources over long periods of time to accomplish a computational task.
Challenges
[edit]The HTC community is also concerned with robustness and reliability of jobs over a long-time scale. That is, being able to create a reliable system from unreliable components. This research is similar to transaction processing, but at a much larger and distributed scale.
Some HTC systems, such as HTCondor and PBS, can run tasks on opportunistic resources. It is a difficult problem, however, to operate in this environment. On one hand the system needs to provide a reliable operating environment for the user's jobs, but at the same time the system must not compromise the integrity of the execute node and allow the owner to always have full control of their resources.
High-throughput vs. high-performance vs. many-task
[edit]There are many differences between high-throughput computing, high-performance computing (HPC), and many-task computing (MTC).
HPC tasks are characterized as needing large amounts of computing power for short periods of time, whereas HTC tasks also require large amounts of computing, but for much longer times (months and years, rather than hours and days).[1][2] HPC environments are often measured in terms of FLOPS.
The HTC community, however, is not concerned about operations per second, but rather operations per month or per year. Therefore, the HTC field is more interested in how many jobs can be completed over a long period of time instead of how fast.
As an alternative definition, the European Grid Infrastructure defines HTC as "a computing paradigm that focuses on the efficient execution of a large number of loosely-coupled tasks",[3] while HPC systems tend to focus on tightly coupled parallel jobs, and as such they must execute within a particular site with low-latency interconnects. Conversely, HTC systems are independent, sequential jobs that can be individually scheduled on many different computing resources across multiple administrative boundaries. HTC systems achieve this using various grid computing technologies and techniques.
MTC aims to bridge the gap between HTC and HPC. MTC is reminiscent of HTC, but it differs in the emphasis of using many computing resources over short periods of time to accomplish many computational tasks (i.e. including both dependent and independent tasks), where the primary metrics are measured in seconds (e.g. FLOPS, tasks/s, MB/s I/O rates), as opposed to operations (e.g. jobs) per month. MTC denotes high-performance computations comprising multiple distinct activities, coupled via file system operations.
See also
[edit]References
[edit]- ^ Beck, Alan (1997-06-27). "High Throughput Computing: An Interview with Miron Livny (Broken link)". HPCWire.
- ^ "High Throughput Computing: An Interview with Miron Livny (Backup link)".
- ^ "EGI Glossary V1".