欢迎来到麦多课文档分享! | 帮助中心 海量文档,免费浏览,给你所需,享你所想!
麦多课文档分享
全部分类
  • 标准规范>
  • 教学课件>
  • 考试资料>
  • 办公文档>
  • 学术论文>
  • 行业资料>
  • 易语言源码>
  • ImageVerifierCode 换一换
    首页 麦多课文档分享 > 资源分类 > PPT文档下载
    分享到微信 分享到微博 分享到QQ空间

    The World Wide Telescope a Digital Library Prototype.ppt

    • 资源ID:373122       资源大小:1.42MB        全文页数:32页
    • 资源格式: PPT        下载积分:2000积分
    快捷下载 游客一键下载
    账号登录下载
    微信登录下载
    二维码
    微信扫一扫登录
    下载资源需要2000积分(如需开发票,请勿充值!)
    邮箱/手机:
    温馨提示:
    如需开发票,请勿充值!快捷下载时,用户名和密码都是您填写的邮箱或者手机号,方便查询和重复下载(系统自动生成)。
    如需开发票,请勿充值!如填写123,账号就是123,密码也是123。
    支付方式: 支付宝扫码支付    微信扫码支付   
    验证码:   换一换

    加入VIP,交流精品资源
     
    账号:
    密码:
    验证码:   换一换
      忘记密码?
        
    友情提示
    2、PDF文件下载后,可能会被浏览器默认打开,此种情况可以点击浏览器菜单,保存网页到桌面,就可以正常下载了。
    3、本站不支持迅雷下载,请使用电脑自带的IE浏览器,或者360浏览器、谷歌浏览器下载即可。
    4、本站资源下载后的文档和图纸-无水印,预览文档经过压缩,下载后原文更清晰。
    5、试题试卷类文档,如果标题没有明确说明有答案则都视为没有答案,请知晓。

    The World Wide Telescope a Digital Library Prototype.ppt

    1、The World Wide Telescope a Digital Library Prototype,Jim Gray, Microsoft Research Alex Szalay, Johns Hopkins University,Talk at OCLC Dublin, OH, 17 May 2004 http:/ Model of Library Science ,Alexandria Gutenberg(Melvil) Dewey Decimal MARC (Henriette Avram) Dublin Core,Yes, I know there have been othe

    2、r things.,Dublin Core,Elements Title Creator Subject Description Publisher Contributor Date Type Format Identifier Source Language Coverage Rights,Elements+ Audience Alternative TableOfContents Abstract Created Valid Available Issued Modified Extent Medium IsVersionOf HasVersion IsReplacedBy Replace

    3、s IsRequiredBy Requires IsPartOf HasPart IsReferencedBy References IsFormatOf HasFormat ConformsTo Spatial Temporal Mediator DateAccepted DateCopyrighted DateSubmitted EducationalLevel AccessRights BibliographicCitation,Encoding LCSH (Lb. Congress Subject Head) MESH (Medical Subject Head) DDC (Dewey

    4、 Decimal Classification) LCC (Lb. Congress Classification) UDC (Universal Decimal Classification) DCMItype (Dublin Core Meta Type) IMT (Internet Media Type) ISO639-2 (ISO language names) RFC1766 (Internet Language tags) URI (Uniform Resource Locator) Point (DCMI spatial point) ISO3166 (ISO country c

    5、odes) Box (DCMI rectangular area) TGN (Getty Thesaurus of Geo Names) Period (DCMI time interval) W3CDTF (W3C date/time) RFC3066 (Language dialects)Types Collection Dataset Event Image InteractiveResouce Service Software Sound Text PhysicalObject StillImage MovingImage,Thanks!,Whats Happening?,We are

    6、 drowning in information Single fixed hierarchy is hopeless Cant organize/find things in a simple tree HOPE: “schematized storage” Objects have “Dublin-like” facets Most facets acquired automatically (email, photo, doc,) Users add annotations and relationships Librarians call this accession Automate

    7、 accession as much as possible Folders/directories are standing queries Organization is “search based” demo sis. Interesting (public) research projects Stuff Ive Seen: http:/ MyLifebits: http:/ Longhorn product embraces & extends these ideas.,The World Wide Telescope a Digital Library Prototype,Jim

    8、Gray, Microsoft Research Alex Szalay, Johns Hopkins University,Talk at OCLC Dublin, OH, 17 May 2004 http:/ what about the talk I promised you?,The Talk,Libraries morphing to integrated text + data (you know that) Dublin Core is bedrock, but many issues remain. (you know that) WWT: All Astronomy data

    9、 and literature online and integrated Problems Librarians have grappled with for centuries: curation, preservation, indexing, access, summarization. Overview of the World-Wide Telescope as a digital library Focus on metadata, schema, curation, and preservation Candidly, we have more problems than so

    10、lutions, but the data is arriving and we are doing the best we can.,New Science Paradigms,Thousand years ago: science was empiricaldescribing natural phenomena Last few hundred years: theoretical branchusing models, generalizations Last few decades: a computational branchsimulating complex phenomena

    11、 Today: data exploration (eScience)synthesizing theory, experiment and computation with advanced data management and statistics,The Big Picture,Experiments & Instruments,Simulations,facts,facts,answers,questions,Data ingest Managing a petabyte Common schema How to organize it? How to reorganize it H

    12、ow to coexist with others,Data Query and Visualization tools Support/training Performance Execute queries in a minute Batch (big) query scheduling,?,The Big Problems,Literature,Other Archives,facts,facts,The Virtual Observatory,Premise: most data is (or could be online) The Internet is the worlds be

    13、st telescope: It has data on every part of the sky In every measured spectral band: optical, x-ray, radio As deep as the best instruments (2 years ago). It is up when you are up The “seeing” is always great Its a smart telescope: links objects and data to literature Software is the capital expense S

    14、hare, standardize, reuse,Why Is Astronomy Special?,Almost all literature online and public ADS: http:/adswww.harvard.edu/ CDS: http:/cdsweb.u-strasbg.fr/Data has no commercial valueNo privacy concerns, freely share results with othersGreat for experimenting with algorithmsIt is real and well documen

    15、tedHigh-dimensional (with confidence intervals)Spatial, temporalDiverse and distributedMany different instruments from many different places and many different timesThe community wants to share the dataThere is a lot of it (soon petabytes),IRAS 100m,ROSAT keV,DSS Optical,2MASS 2m,IRAS 25m,NVSS 20cm,

    16、WENSS 92cm,GB 6cm,Like all sciences, Astronomy Faces an Information Avalanche,Astronomers have a few hundred TB now 1 pixel (byte) / sq arc second 4TB Multi-spectral, temporal, 1PB They mine it looking for new (kinds of) objects or more of interesting ones (quasars), density variations in 400-D spac

    17、e correlations in 400-D space Data doubles every year Data is public after 1 year So, 50% of the data is public Same access for everyone,Publishing Data,Exponential growth: Projects last at least 3-5 years Data sent upwards only at the end of the project Data will never be centralized More responsib

    18、ility on projects Becoming Publishers and Curators Data will reside with projects Analyses must be close to the data,How to Publish Data: Web Services,Web SERVER: Given a url + parameters Returns a web page (often dynamic) Web SERVICE: Given a XML document (soap msg) Returns an XML document (with sc

    19、hema) Tools make this look like an RPC. F(x,y,z) returns (u, v, w) Distributed objects for the web. + naming, discovery, security, Internet-scale distributed computing,Your program,Data In your address space,Web Service,soap,object in xml,Your program,Web Server,http,Web page,The Core Problem: No Ec

    20、onomic Model,The archive user has not yet been born. How can he pay you to curate the data? Q: The Scientist gathered data for his own purpose. Why should he pay (invest time) for your needs? A: thats the scientific method Curating data (documenting the design, the acquisition, and the processing) i

    21、s very hard and there is no reward for doing it. Results are rewarded, not the process of getting them. Storage/archive NOT the problem (its almost free) Curating/Publishing is expensive. Better standards & tools lower costs,Data Inflation Data Pyramid,Level 1A Grows 5TB pixels/year growing to 25TB

    22、2 TB/y compressed growing to 13TB 4 TB today (level 1A in NASA terms),Level 2 Derived data products 10x smaller But there are many catalogs. Publish new edition each year Fixes bugs in data. Must preserve old editions Creates data pyramid Store each edition 1, 2, 3, 4 N N2 bytes Net: Data Inflation:

    23、 L2 L1,What SDSS is Doing: Capture the Bits,Best-effort documenting data and process. Publishing data: often by UPS ( 5TB today and so 5k$ for a copy) Replicating data on 3 continents. EVERYTHING online (tape data is dead data) Archiving all email, discussions, . Keeping all web-logs. Now we need to

    24、 figure out how to organize/search all this metadata.,Making Discoveries,Where are discoveries made? At the edges and boundaries Going deeper, collecting more data, using more colors. Metcalfes law: quadratic benefit Utility of computer networks grows as the number of possible connections: O(N2) Dat

    25、a Federation: quadratic benefit Federation of N archives has utility O(N2) Possibilities for new discoveries grow as O(N2) Current sky surveys have proven this Very early discoveries from SDSS, 2MASS, DPOSS,Federation,Global Federations,Massive datasets live near their owners: Near the instruments s

    26、oftware pipeline Near the applications Near data knowledge and curation Each Archive publishes a (web) service Schema: documents the data Methods on objects (queries) Scientists get “personalized” extracts Uniform access to multiple Archives A common global schema,Schema (aka metadata),Everyone star

    27、ts with the same schema Then the start arguing about semantics. Virtual Observatory: http:/ Metadata based on Dublin Core: http:/ Universal Content Descriptors (UCD): http:/vizier.u-strasbg.fr/doc/UCD.htx Captures quantitative concepts and their units Reduced from 100,000 tables in literature to 1,0

    28、00 terms VOtable a schema for answers to questions http:/www.us-vo.org/VOTable/ Common Queries: Cone Search and Simple Image Access Protocol, SQL Registry: http:/ still a work in progress.,Data Access is Hitting a Wall,You can GREP 1 MB in a second You can GREP 1 GB in a minute You can GREP 1 TB in

    29、2 days You can GREP 1 PB in 3 yearsOh!, and 1PB 4,000 disksAt some point you need indices to limit search parallel data search and analysis This is where databases can help,You can FTP 1 MB in 1 sec You can FTP 1 GB / min (= 1 $/GB) You can FTP 1 TB in 2 days and 1K$ You can FTP 1 PB in 3 years and

    30、1M$,Current practice of data download (FTP/GREP) will not scale to petabyte datasets,Smart Data,Better Data Schemas There is too much data to move around Do data manipulations at database Build custom procedures and functions into DB Unify data Access & Analysis Examples Temporal and spatial indexin

    31、g Pixel processing Automatic parallelism Auto (re)organize Scalable to Petabyte datasets,Move Mohamed to the mountain, not the mountain to Mohamed.,Next-Generation Data Analysis,Looking for Needles in haystacks the Higgs particle Haystacks: dark matter, dark energy, turbulence, ecosystem dynamics Ne

    32、edles are easier than haystacks Global statistics have poor scaling Correlation functions are N2, likelihood techniques N3 As data and computers grow at Moores Law, we can only keep up with N logN A way out? Relax optimal notion (data is fuzzy, answers are approximate) Dont assume infinite computati

    33、onal resources or memory Requires combination of statistics & computer science,The Sloan Digital Sky Survey,Goal Create the most detailed map of the Northern Sky to-date 2.5m telescope 3 degree field of view Two surveys in one 5-color images of of the sky Spectroscopic survey of a million galaxies a

    34、nd quasars Very high data volume 40 Terabytes of raw data 10 Terabytes processed All data public,The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University University of Pittsburgh Fermi National Accelerator Laboratory US Nava

    35、l Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, HeidelbergSloan Foundation, NSF, DOE, NASA,SkyServer,A multi-terabyte database An educational website More than 50 hours of educational exercises Background on astronomy Tutorials and documentation Searc

    36、hable web pages Easy astronomer access to SDSS data. Prototype eScience lab Interactive visual tools for data exploration,http:/skyserver.sdss.org/,Demo SkyServer,atlas education project Mouse in pixel space Explore an object (record space) Explore literature Explore a set Pose a new question,SkyQue

    37、ry (http:/ Query tool using a set of web services Many astronomy archives from Pasadena, Chicago, Baltimore, Cambridge (England) Has grown from 4 to 15 archives, now becoming international standardAllows queries like:,SELECT o.objId, o.r, o.type, t.objIdFROM SDSS:PhotoPrimary o, TWOMASS:PhotoPrimary

    38、 tWHERE XMATCH(o,t)2,Demo SkyQuery Structure,Each SkyNode publishes Schema Web Service Database Web Service,Portal is Plans Query (2 phase) Integrates answers Is itself a web service,MyDB: eScience Workbench,Prototype of bringing analysis to the data Everybody gets a workspace (database) Executes an

    39、alysis at the data Store intermediate results there Long queries run in batch Results shared within groups Only fetch the final results Extremely successful matches work patterns,National Center Biotechnology Information (NCBI) A Better Example,Pubmed: Abstracts and books and Genbank: All Gene seque

    40、nces deposited BLAST and other searches Website to explore data and literature Entrez: unifies many databases with literature (books, journals,) Organizes the data,The Big Picture,Experiments & Instruments,Simulations,facts,facts,answers,questions,Data ingest Managing a petabyte Common schema How to

    41、 organize it? How to reorganize it How to coexist with others,Query and Vis tools Support/training Performance Execute queries in a minute Batch query scheduling,?,The Big Problems,Literature,Other Archives,facts,facts,The Talk,Libraries morphing to integrated text + data (you know that) Dublin Core

    42、 is bedrock, but many issues remain. (you know that) WWT: All Astronomy data and literature online and integrated Problems Librarians have grappled with for centuries: curation, preservation, indexing, access, summarization. Overview of the World-Wide Telescope as a digital library Focus on metadata

    43、, schema, curation, and preservation Candidly, we have more problems than solutions, but the data is arriving and we are doing the best we can.,Education,Educational Projects, aimed at advanced high school students, but covering middle school Teach how to analyze data, discover patterns, not just astronomy 3.7 million project hits, 1.25 million page views of educational content More than 4000 textbooks On the whole web site: 44 million web hits Largely a volunteer effort by many individuals Matches the 2020 curriculum,


    注意事项

    本文(The World Wide Telescope a Digital Library Prototype.ppt)为本站会员(Iclinic170)主动上传,麦多课文档分享仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对上载内容本身不做任何修改或编辑。 若此文所含内容侵犯了您的版权或隐私,请立即通知麦多课文档分享(点击联系客服),我们立即给予删除!




    关于我们 - 网站声明 - 网站地图 - 资源地图 - 友情链接 - 网站客服 - 联系我们

    copyright@ 2008-2019 麦多课文库(www.mydoc123.com)网站版权所有
    备案/许可证编号:苏ICP备17064731号-1 

    收起
    展开