欢迎来到麦多课文档分享! | 帮助中心 海量文档,免费浏览,给你所需,享你所想!
麦多课文档分享
全部分类
  • 标准规范>
  • 教学课件>
  • 考试资料>
  • 办公文档>
  • 学术论文>
  • 行业资料>
  • 易语言源码>
  • ImageVerifierCode 换一换
    首页 麦多课文档分享 > 资源分类 > PPT文档下载
    分享到微信 分享到微博 分享到QQ空间

    The Scalable Checkpoint-Restart Library (SCR)- Overview and.ppt

    • 资源ID:373102       资源大小:795.93KB        全文页数:26页
    • 资源格式: PPT        下载积分:2000积分
    快捷下载 游客一键下载
    账号登录下载
    微信登录下载
    二维码
    微信扫一扫登录
    下载资源需要2000积分(如需开发票,请勿充值!)
    邮箱/手机:
    温馨提示:
    如需开发票,请勿充值!快捷下载时,用户名和密码都是您填写的邮箱或者手机号,方便查询和重复下载(系统自动生成)。
    如需开发票,请勿充值!如填写123,账号就是123,密码也是123。
    支付方式: 支付宝扫码支付    微信扫码支付   
    验证码:   换一换

    加入VIP,交流精品资源
     
    账号:
    密码:
    验证码:   换一换
      忘记密码?
        
    友情提示
    2、PDF文件下载后,可能会被浏览器默认打开,此种情况可以点击浏览器菜单,保存网页到桌面,就可以正常下载了。
    3、本站不支持迅雷下载,请使用电脑自带的IE浏览器,或者360浏览器、谷歌浏览器下载即可。
    4、本站资源下载后的文档和图纸-无水印,预览文档经过压缩,下载后原文更清晰。
    5、试题试卷类文档,如果标题没有明确说明有答案则都视为没有答案,请知晓。

    The Scalable Checkpoint-Restart Library (SCR)- Overview and.ppt

    1、Kathryn Mohror,The Scalable Checkpoint/Restart Library (SCR): Overview and Future Directions,Increased component count in supercomputers means increased failure rate,Todays supercomputers experience failures on the order of hours Future systems are predicted to have failures on the order of minutes

    2、Checkpointing: periodically flush application state to a file Parallel file system (PFS) Bandwidth from cluster to PFS at LLNL: 10s GB/s 100s TB to 1-2 PB of storage Checkpoint data size varies 100s GB to TB,Writing checkpoints to the parallel file system is very expensive,Parallel File System,Hera,

    3、Atlas,Zeus,Gateway Nodes,Compute Nodes,Network Contention,Contention for Shared File System Resources,Contention from Other Clusters for File System,Failures cause loss of valuable compute time,BG/L at LLNL 192K cores Checkpoint every 7.5 hours Achieved 4 days of computation in 6.5 days Atlas at LLN

    4、L 4096 cores Checkpoint every 2 hours 20 - 40 minutes MTBF 4 hours Juno at LLNL 256 cores Average 20 min checkpoints 25% time spent in checkpointing,Node-local storage can be utilized to reduce checkpointing costs,Observations: Only need the most recent checkpoint data. Typically just a single node

    5、failed at a time. Idea: Store checkpoint data redundantly on compute cluster; only write a few checkpoints to parallel file system. Node-local storage is a performance opportunity AND challenge+ Scales with rest of system- Fails and degrades over time- Physically distributed- Limited resource,SCR wo

    6、rks for codes that do globally-coordinated application-level checkpointing,int main(int argc, char* argv) MPI_Init(argc, argv);for(int t = 0; t TIMESTEPS; t+)/* . Do work . */checkpoint();MPI_Finalize();return 0; ,void checkpoint() int rank;MPI_Comm_rank(MPI_COMM_WORLD, ,SCR works for codes that do

    7、globally-coordinated application-level checkpointing,int main(int argc, char* argv) MPI_Init(argc, argv);SCR_Init();for(int t = 0; t TIMESTEPS; t+)/* . Do work . */int flag;SCR_Need_checkpoint( ,void checkpoint() SCR_Start_checkpoint();int rank;MPI_Comm_rank(MPI_COMM_WORLD, ,SCR utilizes node-local

    8、storage and the parallel file system,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_c

    9、heckpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,X,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR uses multiple checkpoint levels for performance and r

    10、esiliency,Aggregate checkpoint bandwidth to node-local storage scales linearly on Coastal,Parallel file system built for 10GB/s,Speedups achieved using SCR with PF3d,SCR can recover from 85% of failures using checkpoints that are 100-1000x faster than PFS,Level 1: Local checkpoint sufficient,42 Temp

    11、orary parallel file system write failure (subsequent job in same allocation succeeded),10 Job hang,7 Transient processor failure(floating-point exception or segmentation fault),Level 2: Partner / XOR checkpoint sufficient,104 Node failure(bad power supply, failed network card, or unexplained reboot)

    12、,Level 3: PFS checkpoint sufficient,23 Permanent parallel file system write failure (no job in same allocation succeeded),3 Permanent hardware failure (bad CPU or memory DIMM),2 Power breaker shut off,Observed 191 failures spanning 5.6 million node hours from 871 runs of PF3d on 3 different clusters

    13、 (Coastal, Hera, and Atlas).,31%,54%,15%,Create a model to estimate the best parameters for SCR and predict its performance on future machines,Several parameters determine SCRs performance: Checkpoint interval Checkpoint types and frequency, e.g. how many local checkpoints between each XOR checkpoin

    14、t Checkpoint costs Failure rates Developed a probabilistic Markov model Metrics Efficiency: How much time is spent actually progressing the simulation Accounts for time spent checkpointing, recovering, and recomputing Parallel file system load: Expected frequency of checkpoints to the parallel file

    15、system,How does checkpointing interval affect efficiency?,C: Checkpoint Cost F: Failure Rate 1x: Todays Values,Maximum efficiency depends on checkpoint cost and failure rates,How does multi-level checkpointing compare to single-level checkpointing to the PFS?,Todays Cost,PFS Checkpoint Cost, Levels,

    16、Multi-level checkpointing requires less writes to the PFS,Todays Cost,More expensive checkpoints are rarer,Higher failure rates require more frequent checkpoints,Multi-level checkpointing requires fewer writes to parallel file system,Todays Failure Rate,Expected Time Between Checkpointing to PFS (se

    17、conds),PFS Checkpoint Cost, Levels,Summary,Multi-level checkpointing library, SCR Low-cost checkpointing schemes up to 1000x faster than PFS Failure analysis of several HPC systems 85% of failures can be recovered from low-cost checkpoints Hierarchical Markov Model that shows benefits of multi-level

    18、 checkpointing: Increased machine efficiency Reduced load on the parallel file system Advantages are expected to increase on future systems. Can still achieve 85% efficiency on 50x less reliable systems,Current and future directions - Theres still more work to do!,Parallel File System,Contention,Use

    19、 an overlay network (MRNet) to write checkpoints to the PFS in a controlled way,Parallel File System,“Forest” of writers,Average Total I/O Time per checkpoint with and without SCR/MRNet,Single writer Every checkpoint to the parallel file system,SCR/MRNet Integration,Still work to do for performance

    20、Current asynchronous drain uses a single writer Forest Although I/O time is greatly improved, theres a scalability problem in SCR_Complete_checkpoint Current implementation uses a single writer and takes too long to drain the checkpoints at larger scales,Compress checkpoints to reduce checkpointing

    21、overheads,Parallel File System,A0=,A1=,A2=,A3=,Partition array A,Interleave array A,Compress array A,70% reduction in checkpoint file size!,Comparison of N-N and N-M Checkpointing,Summary of Compression Effectiveness,Comp Factor = (uncompressed compressed) / compressed * 100,The MRNet nodes add extr

    22、a levels of resiliency,Parallel File System,Geographically disperse nodes in an XOR set for increased resiliency,X,Thanks!,Adam Moody, Greg Bronevetsky, Bronis de Supinski (LLNL) Tanzima Islam, Saurabh Bagchi, Rudolf Eigenmann (Purdue) For more information kathrynllnl.gov Open source, BSD license: http:/ Adam Moody, Greg Bronevetsky, Kathryn Mohror, Bronis R. de Supinski, “Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System,“ LLNL-CONF-427742, SC10.,


    注意事项

    本文(The Scalable Checkpoint-Restart Library (SCR)- Overview and.ppt)为本站会员(syndromehi216)主动上传,麦多课文档分享仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对上载内容本身不做任何修改或编辑。 若此文所含内容侵犯了您的版权或隐私,请立即通知麦多课文档分享(点击联系客服),我们立即给予删除!




    关于我们 - 网站声明 - 网站地图 - 资源地图 - 友情链接 - 网站客服 - 联系我们

    copyright@ 2008-2019 麦多课文库(www.mydoc123.com)网站版权所有
    备案/许可证编号:苏ICP备17064731号-1 

    收起
    展开