爬了10000张NASA关于火星探索的图片,我发现了一个秘密

## 前言 最近,我使用爬虫技术,爬取了美国航空航天局,也就是你电影里经常见到的 `NASA`, 火星探索的相关图片,有 `10000` 张吧。 嗯嗯,小事情,小事情。 完事儿之后,有点小激动,于是就有了这篇文章,将有以下内容: 1. 我为什么要爬取NASA的图片 2. 我是如何爬取NASA图片的(超详细) 3. 我得到了什么(高清大图) 3. 我发现了什么秘密(超劲爆) ## 我为什么要爬NASA的图片 我已经超过35了,瑟瑟发抖啥时候被开了。 天天想着万一哪天失业了干点啥,想着玩个自媒体吧,天天给大家瞎白话。白话点历史谜团,宇宙奥秘啥的,于是我就盯上了 `NASA`。 `NASA` 有各种宇宙探索任务,并且有相关的文章,访谈,图片,视频公开,这是不可多得的资源库。 ## 我是如何爬取NASA图片的(超详细) `NASA` 的网站是可以公开访问的,地址是 ```url https://www.nasa.gov/ ``` ![微信图片_20210630130517.png](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/63470a6f363044a391ab2cc1b65465e0~tplv-k3u1fbpfcp-watermark.image) 打开以后,它首页长这样,可以看到各种内容。右上角还有个搜索框,我们输入 `Mars` 也就是 `火星` 稍等片刻,会展示出与 `Mars` 相关的各种内容,其中有一项 `Mars Exploration` 也就是 `火星探索` 点开之后,就到了一个新页面,然后找到 `Images` 图片,就到了我们爬取的目标页 ```url https://www.nasa.gov/mission_pages/mars/images/index.html ``` ![微信图片_20210630130632.png](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/601a744e6aa1498bb51c626be4d4d3f5~tplv-k3u1fbpfcp-watermark.image) 页面下拉,你将会看到有个大大的按钮,上面写着 `MORE IMAGES`, 点击试试你就发现: 页面的内容,并不是页面直接加载的,而是通过 `api` 请求后,异步渲染的 F12, 打开浏览器开发者模式,重新执行刚才的步骤,观察请求信息,发现都会有如下的情况 ![微信图片_20210630123102.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/1e9e8663b68045208f5a187155004a44~tplv-k3u1fbpfcp-watermark.image) 看上去这个`url`地址非常重要,我们先看他的请求地址: ``` https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi ``` 注意看里面的参数 ``` size=24&from=24 ``` 很显然,`size` 就是每次请求图片的数量,`from` 经过试验,是查询初始位置,我们可以改它来获取其他内容 我们再来看看它的返回信息: ```json { "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 659, "max_score": null, "hits": [{ "_index": "nasa-public", "_type": "ubernode", "_id": "450040", "_score": null, "_source": { "image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ", "topics": ["3140", "3152"], "nid": "450040", "title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "type": "ubernode", "uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision", "collections": ["4525", "5246"], "link-or-attachment": "link", "missions": ["6336"], "primary-tag": "6336", "cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "promo-date-time": "2019-08-05T17:49:00-04:00", "secondary-tag": "3140", "master-image": { "fid": "603128", "alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "width": "1600", "id": "603128", "title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "uri": "public://thumbnails/image/pia23314-16.jpg", "height": "900" }, "ubernode-type": "image" }, "sort": [1565041740000] }, { "_index": "nasa-public", "_type": "ubernode", "_id": "433172", "_score": null, "_source": { "image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.", "topics": ["3152"], "nid": "433172", "title": "Opportunity Emerges in a Dusty Picture", "type": "ubernode", "uri": "/image-feature/opportunity-emerges-in-a-dusty-picture", "collections": ["7628"], "link-or-attachment": "link", "missions": ["3639"], "primary-tag": "3152", "cardfeed-title": "Opportunity Emerges in a Dusty Picture", "promo-date-time": "2018-09-26T12:39:00-04:00", "secondary-tag": "7628", "master-image": { "fid": "584263", "alt": "NASA's Opportunity rover appears as a blip in the center of this square", "width": "1400", "id": "584263", "title": "NASA's Opportunity rover appears as a blip in the center of this square", "uri": "public://thumbnails/image/pia22549-16.jpg", "height": "788" }, "ubernode-type": "image" }, "sort": [1537979940000] }] } } ``` 上面的 `json` 内容太长,我删除了一些重复的,实际上 `hits` 这个数组,也是24个,和页面上显示的图片数量是一样的。那基本可以断定,页面上的信息,就是从这个数组来的。 进而对比发现,`master-image`字段下,就是我们需要的信息,包括 `图片地址`, `图片尺寸`, `图片标题`。 下面就上代码,组装请求URL, 获取内容,下载图片 三步走 我使用了 `Dart` 语言,你们随意 ```dart import 'dart:convert'; import 'package:dio/dio.dart'; main() async { // 每页数量是固定24个,改变初始值即可 for (int from = 0; from < 24 * 100; from = from + 24) { await getPage(from); } } //获取每一页的信息并且下载 Future getPage(int from) async { String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' + from.toString() + '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi'; //获取到内容 var res = await Dio().get(url); var map = jsonDecode(res.toString()); (map['hits']['hits'] as List).forEach((element) async { Uri fileUri = Uri.parse(getUri(element)); String savePath = getSavePath(element); await Dio().downloadUri(fileUri, savePath); print('已下载: ' + savePath); }); } //获取图片下载地址 String getUri(dynamic element) { String uri = element['_source']['master-image']['uri'].toString(); uri = uri.replaceAll('public://', 'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/'); return uri; } //处理信息,并且返回图片保存地址 String getSavePath(dynamic element) { String id = element['_id']; String fid = element['_source']['master-image']['fid'].toString(); String title = element['_source']['master-image']['title'].toString(); String uri = element['_source']['master-image']['uri'].toString(); String savePath = id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last; savePath = savePath.replaceAll('/', ''); savePath = savePath.replaceAll('\\', ''); savePath = savePath.replaceAll('"', ''); savePath = 'images/' + savePath; return savePath; } ``` 上面的代码,还是很简单的,有经验的同学应该一看就懂。 走起来吧。 ``` 已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg 已下载: images/452007_605784_Danielson Crater on Mars.jpg 已下载: images/458478_615132_Gullies on Mars.jpg 已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg 已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg 已下载: images/470381_643473_CME.jpg 已下载: images/458813_615896_Mars.jpg 已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg 已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg 已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg 已下载: images/463659_626874_Avalanche on Mars.jpg 已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg 已下载: images/468636_639726_Mars' Jezero Crater.jpg ``` ## 我得到了什么 这些图片 ![微信图片_20210630121020.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/2a925ff2563c42188d3e0c420dc9fc4f~tplv-k3u1fbpfcp-watermark.image) 还有这些 ![微信图片_20210630131701.png](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b7d75ee4bd0a4ae89d990f21aefc9922~tplv-k3u1fbpfcp-watermark.image) 图片,图片标题都有了, 够看一个月了我估计。 ## 我发现了什么秘密 ![429059_580437_two images of mars, one clear, one fuzzy.png](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d394e2945905472fb4609f4a15cef819~tplv-k3u1fbpfcp-watermark.image) 这张图,我最喜欢了。一个如此清晰,一个如此浑浊,这是为什么? 火星人的裂缝产生器? 好吧,真正的秘密是: `NASA` 的网站竟然没有防采集,不信你也试试。。。
上一篇:【转】【DP_树形DP专辑】【9月9最新更新】【from zeroclock's blog】


下一篇:10000份资料整理 送给你