如何从第三方导出笑话数据到自己的服务器
自己开发的app有个段子模块，是从第三方导入的，用过聚合、点睛、showapi、阿里数据等，后来都有了调用次数限制，最初还开了一年聚合的会员，但是对于非盈利个人开发者，收费还是有点稍高。于是就折腾着将第三方的笑话数据导出来，自己搭了个后台，基于简单方便且免费，选择了bmob。
于是开始导出数据，选用的是最熟悉的java，将多个平台的数据导出。聚合、showapi都是免费的，每天导出的数据量有限，阿里充值了钱，基本可以导出数据平台的所有笑话和趣图。以下是导出数据的代码:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
<span class="hljs-comment">// 聚合数据</span> private <span class="hljs-keyword">static</span> <span class="hljs-built_in">String</span> doGetJuhe(int pageNum, <span class="hljs-built_in">String</span> key) { <span class="hljs-built_in">String</span> url = <span class="hljs-string">"http://v.juhe.cn/joke/content/text.php"</span>; url += <span class="hljs-string">"?page="</span>+pageNum+<span class="hljs-string">"&pagesize=20&key="</span>+key; <span class="hljs-built_in">String</span> result = sendGetRequest(url); <span class="hljs-keyword">return</span> result; } <span class="hljs-comment">// HaoService</span> private <span class="hljs-keyword">static</span> <span class="hljs-built_in">String</span> doGetHaoService(int pageNum,<span class="hljs-built_in">String</span> key) { <span class="hljs-built_in">String</span> url = <span class="hljs-string">"http://apis.haoservice.com/lifeservice/Joke/ContentList"</span>; url += <span class="hljs-string">"?key="</span>+key+<span class="hljs-string">"&pagesize=10&page=+"</span>+pageNum+<span class="hljs-string">"&paybyvas=false"</span>; <span class="hljs-built_in">String</span> result = sendGetRequest(url); <span class="hljs-keyword">return</span> result; } <span class="hljs-comment">/** * 阿里数据 * @param pageNum * @param path "/picJoke" "/gifJoke" "/textJoke" * @return */</span> public <span class="hljs-keyword">static</span> <span class="hljs-built_in">String</span> doGetAli(int pageNum, <span class="hljs-built_in">String</span> path, <span class="hljs-built_in">String</span> appCode) { <span class="hljs-built_in">String</span> host = <span class="hljs-string">"https://ali-joke.showapi.com"</span>; <span class="hljs-comment">//String path = "/picJoke";//"/gifJoke";//"/textJoke";</span> <span class="hljs-built_in">String</span> method = <span class="hljs-string">"GET"</span>; <span class="hljs-built_in">String</span> appcode = appCode; <span class="hljs-built_in">Map</span><<span class="hljs-built_in">String</span>, <span class="hljs-built_in">String</span>> headers = <span class="hljs-keyword">new</span> HashMap<<span class="hljs-built_in">String</span>, <span class="hljs-built_in">String</span>>(); <span class="hljs-comment">//最后在header中的格式(中间是英文空格)为Authorization:APPCODE 83359fd73fe94948385f570e3c139105</span> headers.put(<span class="hljs-string">"Authorization"</span>, <span class="hljs-string">"APPCODE "</span> + appcode); <span class="hljs-built_in">Map</span><<span class="hljs-built_in">String</span>, <span class="hljs-built_in">String</span>> querys = <span class="hljs-keyword">new</span> HashMap<<span class="hljs-built_in">String</span>, <span class="hljs-built_in">String</span>>(); querys.put(<span class="hljs-string">"maxResult"</span>, <span class="hljs-string">"50"</span>); querys.put(<span class="hljs-string">"page"</span>, <span class="hljs-string">""</span> + pageNum); querys.put(<span class="hljs-string">"time"</span>, <span class="hljs-string">"2018-07-19"</span>); <span class="hljs-keyword">try</span> { <span class="hljs-comment">/** * 重要提示如下: * HttpUtils请从 * https://github.com/aliyun/api-gateway-demo-sign-java/blob/master/src/main/java/com/aliyun/api/gateway/demo/util/HttpUtils.java * 下载 * * 相应的依赖请参照 * https://github.com/aliyun/api-gateway-demo-sign-java/blob/master/pom.xml */</span> HttpResponse response = HttpUtils.doGet(host, path, method, headers, querys); <span class="hljs-keyword">return</span> EntityUtils.toString(response.getEntity()); <span class="hljs-comment">//获取response的body</span> <span class="hljs-comment">//System.out.println(EntityUtils.toString(response.getEntity()));</span> } <span class="hljs-keyword">catch</span> (Exception e) { e.printStackTrace(); } <span class="hljs-keyword">return</span> <span class="hljs-literal">null</span>; } |
于是，开始折腾bmob，搭建应用审核后，建一个笑话数据库：主要字段包括 _id(自增id)、title(标题)、content(内容)、createTime(发布时间)、images(图片列表)、author(作者，与user表关联)、type(类型，笑话/趣图/文章...)，其他字段（如feeds，likes，favs，addr，addrName，extra拓展字段等）
然后，需要将之前导出的数据导入到bmob数据库中，bmob支持csv及json文件导入，按照模板格式即可，java代码如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
<span class="hljs-comment">// haoservice</span> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> JSONArray <span class="hljs-title">jsonToHaoService</span><span class="hljs-params">(JSONArray resultJa, String str)</span> </span>{ <span class="hljs-keyword">try</span> { JSONObject jsonO = <span class="hljs-keyword">new</span> JSONObject(str); JSONArray jsonA = jsonO.optJSONArray(<span class="hljs-string">"result"</span>); <span class="hljs-keyword">for</span>(<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < jsonA.length(); i++) { JSONObject jo = jsonA.optJSONObject(i); <span class="hljs-keyword">if</span>(!jo.has(<span class="hljs-string">"content"</span>) || jo.optString(<span class="hljs-string">"content"</span>).length() < <span class="hljs-number">10</span>) { <span class="hljs-keyword">continue</span>; } JSONObject resultJo = <span class="hljs-keyword">new</span> JSONObject(); resultJo.put(<span class="hljs-string">"type"</span>, <span class="hljs-string">"1"</span>); resultJo.put(<span class="hljs-string">"title"</span>, jo.optString(<span class="hljs-string">"title"</span>)); resultJo.put(<span class="hljs-string">"content"</span>, jo.optString(<span class="hljs-string">"content"</span>).replaceAll(<span class="hljs-string">"&#8205;"</span>, <span class="hljs-string">""</span>).replaceAll(<span class="hljs-string">"&zwj;"</span>, <span class="hljs-string">""</span>).replaceAll(<span class="hljs-string">"&hellip;"</span>, <span class="hljs-string">""</span>)); resultJo.put(<span class="hljs-string">"time"</span>, jo.optString(<span class="hljs-string">"updatetime"</span>) + <span class="hljs-string">":00"</span>); <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { resultJa = <span class="hljs-keyword">new</span> JSONArray(); } resultJa.put(resultJo); } } <span class="hljs-keyword">catch</span> (Exception e) { <span class="hljs-comment">// TODO Auto-generated catch block</span> e.printStackTrace(); } <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>; } <span class="hljs-keyword">return</span> resultJa; } <span class="hljs-comment">// 聚合数据</span> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> JSONArray <span class="hljs-title">jsonToJuhe</span><span class="hljs-params">(JSONArray resultJa, String str)</span> </span>{ <span class="hljs-keyword">try</span> { JSONObject jsonO = <span class="hljs-keyword">new</span> JSONObject(str); JSONObject jsonO2 = jsonO.optJSONObject(<span class="hljs-string">"result"</span>); JSONArray jsonA = jsonO2.optJSONArray(<span class="hljs-string">"data"</span>); <span class="hljs-keyword">for</span>(<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < jsonA.length(); i++) { JSONObject jo = jsonA.optJSONObject(i); <span class="hljs-keyword">if</span>(!jo.has(<span class="hljs-string">"content"</span>) || jo.optString(<span class="hljs-string">"content"</span>).length() < <span class="hljs-number">10</span>) { <span class="hljs-keyword">continue</span>; } JSONObject resultJo = <span class="hljs-keyword">new</span> JSONObject(); resultJo.put(<span class="hljs-string">"type"</span>, <span class="hljs-string">"1"</span>); resultJo.put(<span class="hljs-string">"title"</span>, <span class="hljs-string">""</span>); resultJo.put(<span class="hljs-string">"content"</span>, jo.optString(<span class="hljs-string">"content"</span>)); resultJo.put(<span class="hljs-string">"time"</span>, jo.optString(<span class="hljs-string">"updatetime"</span>)); <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { resultJa = <span class="hljs-keyword">new</span> JSONArray(); } resultJa.put(resultJo); } } <span class="hljs-keyword">catch</span> (Exception e) { <span class="hljs-comment">// TODO Auto-generated catch block</span> e.printStackTrace(); } <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>; } <span class="hljs-keyword">return</span> resultJa; } <span class="hljs-comment">// 阿里图片 pic和gif</span> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> JSONArray <span class="hljs-title">jsonToAliImage</span><span class="hljs-params">(JSONArray resultJa, String str)</span> </span>{ <span class="hljs-keyword">try</span> { JSONObject jsonO = <span class="hljs-keyword">new</span> JSONObject(str); JSONObject jsonO2 = jsonO.optJSONObject(<span class="hljs-string">"showapi_res_body"</span>); JSONArray jsonA = jsonO2.optJSONArray(<span class="hljs-string">"contentlist"</span>); <span class="hljs-keyword">for</span>(<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < jsonA.length(); i++) { JSONObject jo = jsonA.optJSONObject(i); <span class="hljs-keyword">if</span>(!jo.has(<span class="hljs-string">"img"</span>) || jo.optString(<span class="hljs-string">"img"</span>).length() < <span class="hljs-number">10</span>) { <span class="hljs-keyword">continue</span>; } JSONObject resultJo = <span class="hljs-keyword">new</span> JSONObject(); resultJo.put(<span class="hljs-string">"type"</span>, <span class="hljs-string">"2"</span>); resultJo.put(<span class="hljs-string">"title"</span>, jo.optString(<span class="hljs-string">"title"</span>)); resultJo.put(<span class="hljs-string">"img"</span>, jo.optString(<span class="hljs-string">"img"</span>)); resultJo.put(<span class="hljs-string">"time"</span>, jo.optString(<span class="hljs-string">"ct"</span>).substring(<span class="hljs-number">0</span>, <span class="hljs-number">19</span>)); <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { resultJa = <span class="hljs-keyword">new</span> JSONArray(); } resultJa.put(resultJo); } } <span class="hljs-keyword">catch</span> (Exception e) { System.out.println(str); <span class="hljs-comment">// TODO Auto-generated catch block</span> e.printStackTrace(); } <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>; } <span class="hljs-keyword">return</span> resultJa; } <span class="hljs-comment">// 阿里文本</span> <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> JSONArray <span class="hljs-title">jsonToAli</span><span class="hljs-params">(JSONArray resultJa, String str)</span> </span>{ <span class="hljs-keyword">try</span> { JSONObject jsonO = <span class="hljs-keyword">new</span> JSONObject(str); JSONObject jsonO2 = jsonO.optJSONObject(<span class="hljs-string">"showapi_res_body"</span>); JSONArray jsonA = jsonO2.optJSONArray(<span class="hljs-string">"contentlist"</span>); <span class="hljs-keyword">for</span>(<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < jsonA.length(); i++) { JSONObject jo = jsonA.optJSONObject(i); <span class="hljs-keyword">if</span>(!jo.has(<span class="hljs-string">"text"</span>) || jo.optString(<span class="hljs-string">"text"</span>).length() < <span class="hljs-number">10</span>) { <span class="hljs-keyword">continue</span>; } JSONObject resultJo = <span class="hljs-keyword">new</span> JSONObject(); resultJo.put(<span class="hljs-string">"type"</span>, <span class="hljs-string">"1"</span>); resultJo.put(<span class="hljs-string">"title"</span>, jo.optString(<span class="hljs-string">"title"</span>)); resultJo.put(<span class="hljs-string">"content"</span>, jo.optString(<span class="hljs-string">"text"</span>)); resultJo.put(<span class="hljs-string">"time"</span>, jo.optString(<span class="hljs-string">"ct"</span>).substring(<span class="hljs-number">0</span>, <span class="hljs-number">19</span>)); <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { resultJa = <span class="hljs-keyword">new</span> JSONArray(); } resultJa.put(resultJo); } } <span class="hljs-keyword">catch</span> (Exception e) { System.out.println(str); <span class="hljs-comment">// TODO Auto-generated catch block</span> e.printStackTrace(); } <span class="hljs-keyword">if</span>(resultJa == <span class="hljs-keyword">null</span>) { <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>; } <span class="hljs-keyword">return</span> resultJa; } |
原文：简书ThinkinLiu 博客: IT老五
ps：这里主要是记录下来自己以后可能用到，可能写得不是太明白，没去整理了
