{"id":1878,"date":"2024-09-13T20:25:36","date_gmt":"2024-09-13T12:25:36","guid":{"rendered":"https:\/\/www.gnn.club\/?p=1878"},"modified":"2024-10-10T14:43:23","modified_gmt":"2024-10-10T06:43:23","slug":"rnn","status":"publish","type":"post","link":"http:\/\/www.gnn.club\/?p=1878","title":{"rendered":"\u5faa\u73af\u795e\u7ecf\u7f51\u7edc\uff08RNN\uff09"},"content":{"rendered":"<h1><img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913202835406.png\" style=\"height:50px;display:inline\"> Deep Learning<\/h1>\n<hr \/>\n<p>create by Arwin Yu<\/p>\n<h2>Tutorial 03  - Recurrent Neural Networks - Sequential Tasks<\/h2>\n<hr \/>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913202904521.png\" style=\"height:200px\">\n<\/p>\n<h3><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/bubbles\/50\/000000\/checklist.png\" style=\"height:50px;display:inline\"> Agenda<\/h3>\n<hr \/>\n<ul>\n<li>\u81ea\u7136\u8bed\u8a00\u5904\u7406\u548c\u5e8f\u5217\u6570\u636e(Natural Language Processing and Sequences)\n<ul>\n<li>\u8bed\u8a00\u6a21\u578b<\/li>\n<li>\u8bcd\u5d4c\u5165<\/li>\n<li>\u4efb\u52a1\u7c7b\u578b<\/li>\n<li>\u6587\u672c\u9884\u5904\u7406<\/li>\n<\/ul>\n<\/li>\n<li>\u5faa\u73af\u795e\u7ecf\u7f51\u7edc(Recurrent Neural Networks)<\/li>\n<li>\u957f\u77ed\u671f\u8bb0\u5fc6\u7f51\u7edc(LSTM)<\/li>\n<li>\u95e8\u63a7\u5faa\u73af\u5355\u5143(GRU)<\/li>\n<li>\u793a\u4f8b\uff08PyTorch RNN Model Example\uff09<\/li>\n<\/ul>\n<h2><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/nolan\/64\/language.png\" style=\"height:50px;display:inline\"> \u81ea\u7136\u8bed\u8a00\u5904\u7406\u548c\u5e8f\u5217\u6570\u636e<\/h2>\n<hr \/>\n<ul>\n<li>\u5e8f\u5217\u5efa\u6a21\u662f\u5efa\u6a21\u5e8f\u5217\u7684\u9886\u57df\uff0c\u4f8b\u5982\u6587\u672c\u53e5\u5b50\u3001\u89c6\u9891\u3001\u80a1\u7968\u4ef7\u683c\u3001\u5f3a\u5316\u5b66\u4e60\u6216\u81ea\u52a8\u9a7e\u9a76\u4e2d\u7684\u8f68\u8ff9\u3001\u5929\u6c14\u9884\u62a5\u7b49\u2026\u2026<\/li>\n<li>\u4e0e\u6211\u4eec\u4e4b\u524d\u5047\u8bbe\u7684\u6570\u636e\u662f\u72ec\u7acb\u540c\u5206\u5e03\u7684\u4e0d\u540c\uff0c\u5e8f\u5217\u4e2d\u901a\u5e38\u5e76\u975e\u5982\u6b64\uff08\u4f8b\u5982\uff0c\u5982\u679c\u4f60\u968f\u673a\u6539\u53d8\u53e5\u5b50\u4e2d\u7684\u5355\u8bcd\uff0c\u5c31\u5f88\u96be\u7406\u89e3\u5b83\u7684\u542b\u4e49\uff09\u3002<\/li>\n<li>\u6211\u4eec\u5c06\u91cd\u70b9\u5173\u6ce8\u81ea\u7136\u8bed\u8a00\u5904\u7406 (NLP) \u9886\u57df\u7684\u6587\u672c\u6570\u636e\u3002<\/li>\n<\/ul>\n<h4>\u8bed\u8a00\u6a21\u578b\uff08Language Models\uff09<\/h4>\n<hr \/>\n<p>\u8bed\u8a00\u6a21\u578b\u662f\u4e00\u79cd\u7528\u6765\u8bc4\u4f30\u4e00\u6bb5\u6587\u672c\u51fa\u73b0\u6982\u7387\u7684\u5de5\u5177\u3002\u4f8b\u5982\uff0c\u7ed9\u5b9a\u4e00\u6bb5\u6587\u672c\u201c\u6211\u559c\u6b22\u4eba\u5de5\u667a\u80fd\u201d\uff0c\u8bed\u8a00\u6a21\u578b\u53ef\u4ee5\u8ba1\u7b97\u51fa\u8fd9\u6bb5\u6587\u672c\u7684\u6982\u7387\uff0c\u5373$p(\u6211, \u559c\u6b22, \u4eba\u5de5, \u667a\u80fd)$\u3002<\/p>\n<p>\u4e3a\u4e86\u7b80\u5316\u8ba1\u7b97\uff0c\u6700\u5e38\u89c1\u7684\u65b9\u6cd5\u662f\u5229\u7528\u6982\u7387\u5206\u89e3\u7684\u57fa\u672c\u539f\u5219\u548c\u9a6c\u5c14\u53ef\u592b\u5047\u8bbe\uff0c\u5c06\u590d\u6742\u7684\u6982\u7387\u5206\u5e03\u62c6\u89e3\u6210\u591a\u4e2a\u6761\u4ef6\u6982\u7387\u7684\u4e58\u79ef\uff0c<\/p>\n<p>\u5373<br \/>\np(\u6211,\u559c\u6b22,\u4eba\u5de5,\u667a\u80fd)=p(\u6211)p(\u559c\u6b22\u2223\u6211)p(\u4eba\u5de5\u2223\u559c\u6b22)p(\u667a\u80fd\u2223\u4eba\u5de5)\u3002\u8fd9\u6837\uff0c\u8ba1\u7b97\u6bcf\u4e2a\u8bcd\u7684\u51fa\u73b0\u6982\u7387\u53ea\u9700\u8981\u8003\u8651\u5b83\u524d\u4e00\u4e2a\u8bcd\u7684\u60c5\u51b5\u3002<\/p>\n<p>\u7136\u800c\uff0c\u8fd9\u79cd\u65b9\u6cd5\u5b58\u5728\u4e00\u4e9b\u4e0d\u5fc5\u8981\u7684\u5047\u8bbe\u3002\u9a6c\u5c14\u53ef\u592b\u5047\u8bbe\u8ba4\u4e3a\u6bcf\u4e2a\u8bcd\u7684\u51fa\u73b0\u53ea\u4f9d\u8d56\u4e8e\u524d\u4e00\u4e2a\u8bcd\uff0c\u800c\u5ffd\u7565\u4e86\u6574\u4e2a\u53e5\u5b50\u7684\u5386\u53f2\u4fe1\u606f\u3002\u5b9e\u9645\u4e0a\uff0c\u67d0\u4e2a\u8bcd\u7684\u51fa\u73b0\u53ef\u80fd\u4e0e\u66f4\u65e9\u4e4b\u524d\u7684\u8bcd\u6709\u5173\uff0c\u800c\u4e0d\u4ec5\u4ec5\u662f\u524d\u4e00\u4e2a\u8bcd\u3002\u56e0\u6b64\uff0c\u9a6c\u5c14\u53ef\u592b\u5047\u8bbe\u5728\u67d0\u4e9b\u60c5\u51b5\u4e0b\u53ef\u80fd\u8fc7\u4e8e\u7b80\u5316\uff0c\u65e0\u6cd5\u6355\u6349\u5230\u8bed\u8a00\u7684\u590d\u6742\u6027\u3002<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203042685.gif\" style=\"height:200px\">\n<\/p>\n<ul>\n<li><a href=\"https:\/\/medium.com\/perceptronai\/recurrent-neural-network-an-introduction-for-beginners-1c13a541c906\">Image Source<\/a><\/li>\n<\/ul>\n<h4>\u8bcd\u5d4c\u5165\uff08Word embeddings\uff09<\/h4>\n<hr \/>\n<ul>\n<li>\n<p>\u5b9e\u9645\u4e0a\uff0c\u5b58\u5728\u4e00\u4e9b\u7ecf\u5178\u7684\u673a\u5668\u5b66\u4e60\u65b9\u6cd5\u6216\u6570\u5b66\u6a21\u578b\u6765\u6784\u5efa\u8bed\u8a00\u6a21\u578b\uff0c\u4f46\u672c\u8bfe\u7a0b\u7684\u91cd\u70b9\u662f\u6df1\u5ea6\u5b66\u4e60\uff0c\u56e0\u6b64\u6211\u4eec\u5c06\u628a\u7ecf\u5178\u65b9\u6cd5\u7559\u7ed9\u5927\u5bb6\u8fdb\u884c\u63a2\u7d22\uff0c\u53ef\u4ee5\u53c2\u8003\u7f51\u4e0a\u5404\u79cd NLP \u8bfe\u7a0b\u3002<\/p>\n<\/li>\n<li>\n<p>\u6211\u4eec\u63d0\u51fa\u7684\u7b2c\u4e00\u4e2a\u95ee\u9898\u662f\uff1a\u5982\u4f55\u5c06\u5b57\u7b26\u4e32\uff08\u5b57\u7b26\/\u5355\u8bcd\uff09\u8f6c\u6362\u4e3a\u53ef\u4ee5\u8f93\u5165\u795e\u7ecf\u7f51\u7edc\u7684\u6570\u5b57\uff1f<\/p>\n<\/li>\n<li>\n<p>\u8fd9\u79cd\u8f6c\u6362\u901a\u5e38\u79f0\u4e3a<strong>\u5d4c\u5165\uff08Embeddings\uff09<\/strong>\u3002<\/p>\n<\/li>\n<li>\n<p>PyTorch \u4e2d\u7684\u5d4c\u5165\u5c42\uff1a<a href=\"https:\/\/pytorch.org\/docs\/stable\/generated\/torch.nn.Embedding.html\"><code>nn.Embedding(num_embeddings, embedding_dim)<\/code><\/a>\u3002<\/p>\n<\/li>\n<li>\n<p>\u6709\u51e0\u79cd\u9884\u5148\u8bad\u7ec3\u597d\u7684\u6a21\u578b\uff0c\u4f8b\u5982 BERT \u6216 Word2Vec\uff0c\u5b83\u4eec\u5df2\u7ecf\u8bad\u7ec3\u4e86\u5177\u6709\u67d0\u4e9b\u76ee\u6807\u7684\u8bcd\u5d4c\u5165\uff0c\u5e76\u5df2\u51c6\u5907\u597d\u7528\u4e8e\u4e0b\u6e38\u4efb\u52a1\u3002<\/p>\n<\/li>\n<\/ul>\n<p>one-hot\u7f16\u7801\u5f62\u5f0f\u7684\u8bcd\u5d4c\u5165\uff1a<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203146354.png\" style=\"height:200px\">\n<\/p>\n<p><a href=\"https:\/\/www.shanelynn.ie\/get-busy-with-word-embeddings-introduction\/\">Image Source<\/a><\/p>\n<p>\u5bf9\u4e8e\u7ed9\u5b9a\u7684\u8bcd $w$ \uff0c\u5d4c\u5165\u77e9\u9635\uff08Embedding Matix\uff09 $E$ \u662f\u4e00\u4e2a\u6743\u91cd\u77e9\u9635\uff08weight Matix\uff09\uff0c\u5b83\u5c06\u8bcd\u7684\u72ec\u70ed (1-hot) \u8868\u793a $o_w$ \u6620\u5c04\u5230\u5176\u5d4c\u5165 $e_w$ \uff0c\u5177\u4f53\u5982\u4e0b:<br \/>\n$$<br \/>\ne_w=E o_w<br \/>\n$$<\/p>\n<p>tips: \u5d4c\u5165\u77e9\u9635\u7684\u5b66\u4e60\u53ef\u4ee5\u901a\u8fc7\u76ee\u6807\/\u4e0a\u4e0b\u6587\u4f3c\u7136\u6a21\u578b\u5b8c\u6210\u3002<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203330361.png\" style=\"height:200px\">\n<\/p>\n<p><a href=\"https:\/\/www.tylercrosse.com\/ideas\/semantic-search\">Image Source<\/a><\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203434842.gif\" style=\"height:200px\">\n<\/p>\n<p><a href=\"https:\/\/lena-voita.github.io\/nlp_course\/word_embeddings.html\">Image Source<\/a><\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203513194.png\" style=\"height:300px\">\n<\/p>\n<h3>Word2vec<\/h3>\n<p>Word2vec \u662f\u4e00\u4e2a\u6846\u67b6\uff0c\u65e8\u5728\u901a\u8fc7\u4f30\u8ba1\u7ed9\u5b9a\u8bcd\u88ab\u5176\u4ed6\u8bcd\u5305\u56f4\u7684\u53ef\u80fd\u6027\u6765\u5b66\u4e60\u8bcd\u5d4c\u5165\u3002\u5e38\u89c1\u7684\u6a21\u578b\u5305\u62ec skip-gram\u3001\u8d1f\u91c7\u6837\uff08negative sampling\uff09\u548c CBOW\uff08\u8fde\u7eed\u8bcd\u888b\u6a21\u578b\uff09\u3002<\/p>\n<p>\u4e0b\u56fe\u4e2d\u5c55\u793a\u4e86 Word2vec \u7684\u57fa\u672c\u5de5\u4f5c\u6d41\u7a0b\uff1a<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913203547275.png\" style=\"height:200px\">\n<\/p>\n<p>\u8bad\u7ec3\u7f51\u7edc\u6267\u884c\u4ee3\u7406\u4efb\u52a1\uff1a\u5de6\u56fe\u663e\u793a\u4e86\u4e00\u4e2a\u7f51\u7edc\u5728\u6267\u884c\u4ee3\u7406\u4efb\u52a1\uff08\u4f8b\u5982\u9884\u6d4b\u4e00\u4e2a\u8bcd\u7684\u4e0a\u4e0b\u6587\u8bcd\uff09\u65f6\u7684\u8bad\u7ec3\u8fc7\u7a0b\u3002\u8f93\u5165\u7684\u662f\u4e00\u4e2a\u53e5\u5b50\u7247\u6bb5\u201c...A cute teddy bear is reading...\u201d\uff0c\u901a\u8fc7\u795e\u7ecf\u7f51\u7edc\u5b66\u4e60\u8be5\u53e5\u5b50\u4e2d\u5404\u8bcd\u7684\u5173\u7cfb\u3002<\/p>\n<p>\u63d0\u53d6\u9ad8\u7ea7\u8868\u793a\uff1a\u4e2d\u56fe\u5c55\u793a\u4e86\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u63d0\u53d6\u5230\u7684\u8bcd\u7684\u9ad8\u7ea7\u8868\u793a\u3002\u8fd9\u4e9b\u8868\u793a\u4fdd\u7559\u4e86\u8bcd\u4e4b\u95f4\u7684\u8bed\u4e49\u5173\u7cfb\u3002<\/p>\n<p>\u8ba1\u7b97\u8bcd\u5d4c\u5165\uff1a\u53f3\u56fe\u5c55\u793a\u4e86\u6700\u7ec8\u5f97\u5230\u7684\u8bcd\u5d4c\u5165\uff0c\u4f8b\u5982\u201cteddy bear\u201d\u4e0e\u201csoft\u201d\u3001\u201cart\u201d\u3001\u201cPersian poetry\u201d\u7b49\u8bcd\u7684\u5173\u7cfb\u3002\u8fd9\u4e9b\u8bcd\u5d4c\u5165\u53ef\u4ee5\u7528\u4e8e\u5404\u79cd\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u3002<\/p>\n<ul>\n<li>Skip-gram<\/li>\n<\/ul>\n<p>Skip-gram Word2vec \u6a21\u578b\u662f\u4e00\u79cd\u6709\u76d1\u7763\u7684\u5b66\u4e60\u4efb\u52a1\uff0c\u901a\u8fc7\u8bc4\u4f30\u7ed9\u5b9a\u76ee\u6807\u8bcd $t$ \u4e0e\u4e0a\u4e0b\u6587\u8bcd $c$ \u540c\u65f6\u51fa\u73b0\u7684\u53ef\u80fd\u6027\u6765\u5b66\u4e60\u8bcd\u5d4c\u5165\u3002\u8bb0 $\\theta_t$\u4e3a\u4e0e\u76ee\u6807\u8bcd $t$ \u76f8\u5173\u7684\u53c2\u6570\uff0c\u76ee\u6807\u8bcd $t$ \u5728\u7ed9\u5b9a\u4e0a\u4e0b\u6587\u8bcd $c$ \u65f6\u7684\u6982\u7387 $P(t \\mid c)$ \u8868\u793a\u5982\u4e0b:<br \/>\n$$<br \/>\nP(t \\mid c)=\\frac{\\exp \\left(\\theta_t^T e_c\\right)}{\\sum_{j=1}^{V \\mid} \\exp \\left(\\theta_j^T e_c\\right)}<br \/>\n$$<\/p>\n<p>CBOW\uff08\u8fde\u7eed\u8bcd\u888b\u6a21\u578b\uff09 \u662f\u53e6\u4e00\u79cd Word2vec \u6a21\u578b\uff0c\u4e0eSkip-gram\u7684\u903b\u8f91\u6b63\u597d\u60f3\u53cd\uff0c\u5b83\u4f7f\u7528\u5468\u56f4\u7684\u8bcd\u6765\u9884\u6d4b\u7ed9\u5b9a\u7684\u8bcd\u3002<\/p>\n<ul>\n<li>Negative sampling <\/li>\n<\/ul>\n<p>\u8d1f\u91c7\u6837\u662f\u4e00\u7ec4\u4f7f\u7528\u903b\u8f91\u56de\u5f52\u7684\u4e8c\u5143\u5206\u7c7b\u5668\uff0c\u65e8\u5728\u8bc4\u4f30\u7ed9\u5b9a\u4e0a\u4e0b\u6587\u8bcd\u548c\u7ed9\u5b9a\u76ee\u6807\u8bcd\u540c\u65f6\u51fa\u73b0\u7684\u53ef\u80fd\u6027\u3002\u6a21\u578b\u5728 $k$ \u4e2a\u8d1f\u6837\u672c\u548c 1 \u4e2a\u6b63\u6837\u672c\u7684\u96c6\u5408\u4e0a\u8fdb\u884c\u8bad\u7ec3\u3002\u7ed9\u5b9a\u4e00\u4e2a\u4e0a\u4e0b\u6587\u8bcd $c$\u548c\u4e00\u4e2a\u76ee\u6807\u8bcd $t \uff0c$ \u9884\u6d4b\u8868\u793a\u5982\u4e0b:<br \/>\n$$<br \/>\nP(y=1 \\mid c, t)=\\sigma\\left(\\theta_t^T e_c\\right)<br \/>\n$$<\/p>\n<ul>\n<li>GloVe <\/li>\n<\/ul>\n<p>\u201cGlobal Vectors for Word Representation\u201d\uff08\u5168\u5c40\u8bcd\u5411\u91cf\u8868\u793a\uff09\u7684\u7b80\u79f0\uff0c\u662f\u4e00\u79cd\u7528\u4e8e\u751f\u6210\u8bcd\u5d4c\u5165\u7684\u65b9\u6cd5\u3002\u5b83\u901a\u8fc7\u4f7f\u7528\u5171\u73b0\u77e9\u9635$X$\u6765\u5b9e\u73b0\uff0c \u5171\u73b0\u77e9\u9635 $X$ \u8bb0\u5f55\u4e86\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u5728\u4e00\u5b9a\u4e0a\u4e0b\u6587\u7a97\u53e3\u5185\u5171\u540c\u51fa\u73b0\u7684\u6b21\u6570\u3002\u4f8b\u5982\uff0c\u5047\u8bbe\u6211\u4eec\u6709\u4e00\u4e2a\u5355\u8bcd&quot;apple&quot;\u548c\u5b83\u7684\u4e0a\u4e0b\u6587\u8bcd&quot;fruit&quot;\uff0c\u5982\u679c\u201capple&quot;\u548c\u201cfruit&quot;\u5728\u5927\u91cf\u6587\u672c\u4e2d\u7ecf\u5e38\u4e00\u8d77\u51fa\u73b0\uff0c\u90a3\u4e48 $X_{i, j}$ \u7684\u503c\u4f1a\u5f88\u9ad8\u3002GloVe \u901a\u8fc7\u5206\u6790\u5927\u91cf\u6587\u672c\u4e2d\u8bcd\u4e0e\u8bcd\u5171\u540c\u51fa\u73b0\u7684\u9891\u7387\uff0c\u6765\u751f\u6210\u6bcf\u4e2a\u8bcd\u7684\u5411\u91cf\u8868\u793a\u3002\u8fd9\u4e9b\u5411\u91cf\u53ef\u4ee5\u6355\u6349\u5230\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u7684\u8bed\u4e49\u5173\u7cfb\uff0c\u5c3d\u7ba1\u6bcf\u4e2a\u5411\u91cf\u7684\u5177\u4f53\u6570\u503c\u53ef\u80fd\u4e0d\u5bb9\u6613\u89e3\u91ca\u5176\u5b9e\u9645\u610f\u4e49\u3002\u5177\u4f53\u800c\u8a00\uff0cGloVe\u7684\u635f\u5931\u51fd\u6570\u8bbe\u8ba1\u5982\u4e0b:<br \/>\n$$<br \/>\nJ=\\sum_{i, j=1}^V f\\left(X_{i, j}\\right)\\left(w_i^T \\cdot \\tilde{w}_j+b_i+\\tilde{b}_j-\\log X_{i, j}\\right)^2<br \/>\n$$<\/p>\n<p>\u5176\u4e2d:<\/p>\n<ul>\n<li>$V$\u662f\u8bcd\u6c47\u8868\u7684\u5927\u5c0f\u3002<\/li>\n<li>$w_i$\u548c$\\tilde{w}_j$\u5206\u522b\u662f\u8bcd$i$\u548c\u8bcd$j$\u7684\u8bcd\u5411\u91cf\u3002<\/li>\n<li>$b_i$\u548c$\\tilde{b}_j$\u662f\u8bcd$i$\u548c\u8bcd$j$\u7684\u504f\u7f6e\u9879\u3002<\/li>\n<li>$\\log X_{i,j}$\u662f\u8bcd$i$\u548c\u8bcd$j$\u5171\u73b0\u6b21\u6570\u7684\u5bf9\u6570\u3002<\/li>\n<li>$f(X_{i,j})$\u662f\u4e00\u4e2a\u6743\u91cd\u51fd\u6570\uff0c\u7528\u6765\u51cf\u5c11\u5171\u73b0\u6b21\u6570\u8fc7\u5c11\u7684\u8bcd\u5bf9\u635f\u5931\u51fd\u6570\u7684\u5f71\u54cd\u3002<\/li>\n<\/ul>\n<h4>Forms of Sequence Prediction Tasks<\/h4>\n<hr \/>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204007572.png\" style=\"height:600px\">\n<\/p>\n<p><a href=\"https:\/\/stanford.edu\/~shervine\/teaching\/cs-230\/cheatsheet-recurrent-neural-networks#\/\">Image Source<\/a><\/p>\n<h2><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/bubbles\/50\/000000\/connection-sync.png\" style=\"height:50px;display:inline\"> Text Preprocessing<\/h2>\n<hr \/>\n<ul>\n<li>\n<p>\u5728\u6df1\u5165\u7814\u7a76\u5177\u4f53\u6a21\u578b\u4e4b\u524d\uff0c\u6211\u4eec\u9700\u8981\u4e86\u89e3\u5982\u4f55\u5904\u7406\u6587\u672c\u6570\u636e\uff0c\u56e0\u4e3a\u4f60\u4e0d\u80fd\u76f4\u63a5\u5c06\u5355\u8bcd\u8f93\u5165\u795e\u7ecf\u7f51\u7edc\uff0c\u8fd8\u9700\u8981\u4e3a\u5b83\u4eec\u63d0\u4f9b\u4e00\u4e9b\u6570\u5b57\u8868\u793a\u3002<\/p>\n<\/li>\n<li>\n<p>\u5728\u7ecf\u5178 NLP \u4e2d\uff0c\u5355\u8bcd\u6709\u65f6\u8868\u793a\u4e3a\u72ec\u70ed\u5411\u91cf\uff0c\u5176\u4e2d\u5411\u91cf\u7684\u5927\u5c0f\u662f\u8bcd\u6c47\u91cf\u3002<\/p>\n<\/li>\n<li>\n<p>\u4e00\u822c\u6b65\u9aa4\u5982\u4e0b\uff1a<\/p>\n<\/li>\n<li>\n<p>\u5c06\u6587\u672c\u4f5c\u4e3a\u5b57\u7b26\u4e32\u52a0\u8f7d\u5230\u5185\u5b58\u4e2d\u3002<\/p>\n<\/li>\n<li>\n<p><strong>\u6807\u8bb0\u5316(Tokenization)<\/strong>\uff1a\u5c06\u5b57\u7b26\u4e32\u62c6\u5206\u4e3atokens\uff08\u4f8b\u5982\uff0c\u5355\u8bcd\u3001\u5355\u8bcd\u7684\u90e8\u5206\u548c\u5b57\u7b26\uff09\u3002<\/p>\n<\/li>\n<li>\n<p><strong>\u8bcd\u6c47\u8868(Vocabulary)<\/strong>\uff1a\u6784\u5efa\u8bcd\u6c47\u8868\u4ee5\u5c06\u62c6\u5206\u7684 token \u6620\u5c04\u5230\u6570\u5b57\u7d22\u5f15\uff0c\u4ee5\u4fbf\u6a21\u578b\u53ef\u4ee5\u8f7b\u677e\u64cd\u4f5c\u5b83\u4eec\u3002<\/p>\n<\/li>\n<li>\n<p>\u6211\u4eec\u5c06\u4f7f\u7528 PyTorch \u5b98\u65b9\u5e93 <a href=\"https:\/\/pytorch.org\/text\/stable\/index.html\"><code>torchtext<\/code><\/a> \u6765\u5904\u7406\u6587\u672c\u6570\u636e\u3002<\/p>\n<\/li>\n<li>\n<p>\u6211\u4eec\u4f7f\u7528 IMDB \u6570\u636e\u96c6\uff1a\u8be5\u6570\u636e\u96c6\u5305\u542b\u6807\u8bb0\u4e3a\u201c\u6b63\u9762\u201d\u548c\u201c\u8d1f\u9762\u201d\uff08\u5206\u522b\u8868\u793a\u597d\u8bc4\u548c\u5dee\u8bc4\uff09\u7684\u7535\u5f71\u8bc4\u8bba\u3002<\/p>\n<\/li>\n<li>\n<p>\u6b64\u4efb\u52a1\u5728 NLP \u4e2d\u79f0\u4e3a<strong>\u60c5\u7eea\u5206\u6790<\/strong>\uff0c\u672c\u8d28\u4e0a\u662f\u4e00\u4e2a\u5206\u7c7b\u4efb\u52a1\u3002<\/p>\n<\/li>\n<li>\n<p>\u5982\u679c\u4f60\u60f3\u52a0\u8f7d\u5176\u4ed6\u6570\u636e\u96c6\u6216\u4ece\u6587\u672c\u6587\u4ef6\u521b\u5efa\u81ea\u5b9a\u4e49\u6570\u636e\u96c6\uff1a<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/pytorch.org\/tutorials\/beginner\/torchtext_custom_dataset_tutorial.html\">https:\/\/pytorch.org\/tutorials\/beginner\/torchtext_custom_dataset_tutorial.html<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/pytorch.org\/text\/stable\/datasets.html\">https:\/\/pytorch.org\/text\/stable\/datasets.html<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/pytorch.org\/data\/beta\/torchdata.datapipes.iter.html#text-datapipes\">https:\/\/pytorch.org\/data\/beta\/torchdata.datapipes.iter.html#text-datapipes<\/a><\/p>\n<\/li>\n<li>\n<p><strong>\u7279\u6b8a\u6807\u8bb0<\/strong>\uff1a<\/p>\n<\/li>\n<li>\n<p><code>&lt;sos&gt;<\/code>\/<code>&lt;bos&gt;<\/code> - \u6807\u8bb0\u53e5\u5b50\u7684<strong>\u5f00\u59cb\/\u5f00\u59cb<\/strong>\u7684\u6807\u8bb0\u3002<br \/>\n*<pad>` - \u7528\u6765<strong>pad<\/strong>\u7684\u53e5\u5b50\u6bd4\u6279\u5904\u7406\u4e2d\u6700\u957f\u7684\u53e5\u5b50\u3002\u5355\u8bcd\uff0c\u6216\u8005\u60a8\u51b3\u5b9a\u9057\u6f0f\u4e00\u4e9b\u5355\u8bcd\uff0c\u4f8b\u5982\u540d\u5b57\uff09\u3002<\/p>\n<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204112528.png\" style=\"height:300px\">\n<\/p>\n<p><a href=\"https:\/\/livebook.manning.com\/book\/natural-language-processing-in-action\/chapter-10\/\">Image Source<\/a><\/p>\n<p>\u53f3\u56fe\u5c55\u793a\u4e86\u901a\u8fc7\u52a8\u6001\u6279\u5904\u7406\u6765\u5904\u7406\u4e0d\u540c\u957f\u5ea6\u7684\u5e8f\u5217\u7684\u65b9\u6cd5\u3002\u867d\u7136\u5728\u6bcf\u4e2a\u6279\u6b21\u5185\u90e8\u8fdb\u884c\u4e86\u586b\u5145\u5bf9\u9f50\uff0c\u4f46\u4e0d\u540c\u6279\u6b21\u4e4b\u95f4\u53ef\u4ee5\u6709\u4e0d\u540c\u7684\u957f\u5ea6\uff0c\u4ece\u800c\u51cf\u5c11\u8ba1\u7b97\u4e2d\u7684\u586b\u5145\u6d6a\u8d39\uff0c\u63d0\u9ad8\u8ba1\u7b97\u6548\u7387\u3002\u540c\u65f6\uff0c\u6279\u5904\u7406\u4e0d\u4f1a\u6539\u53d8\u5e8f\u5217\u5728\u539f\u6587\u4e2d\u7684\u987a\u5e8f\uff0c\u53ea\u662f\u4e3a\u4e86\u8ba1\u7b97\u4f18\u5316\u8fdb\u884c\u7684\u5206\u7ec4\u3002<\/p>\n<h2><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/nolan\/64\/re-enter-pincode.png\" style=\"height:50px;display:inline\"> Recurrent Neural Networks (RNNs)<\/h2>\n<hr \/>\n<ul>\n<li><strong>\u5faa\u73af\u795e\u7ecf\u7f51\u7edc (RNN)<\/strong> \u7684\u7406\u5ff5\uff1a\u4fdd\u5b58\u67d0\u4e00\u65f6\u523b\u7684\u8f93\u51fa\u5e76\u5c06\u5176\u53cd\u9988\u7ed9\u4e0b\u4e00\u65f6\u523b\u7684\u8f93\u5165\u3002<\/li>\n<li>\u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65\u9aa4\u4e2d\uff0c\u6211\u4eec\u90fd\u4f1a\u7ef4\u62a4\u4e00\u4e9b\u72b6\u6001\uff08\u4ece\u4e0a\u4e00\u4e2a\u65f6\u95f4\u6b65\u9aa4\u63a5\u6536\uff09\u2014\u2014<strong>hidden state \u9690\u85cf\u72b6\u6001<\/strong>\uff0c\u5b83\u4ee3\u8868\u6211\u4eec\u8fc4\u4eca\u4e3a\u6b62\u6240\u8bfb\u7684\u5185\u5bb9\u3002\u8fd9\u4e0e\u5f53\u524d\u6b63\u5728\u9605\u8bfb\u7684\u5355\u8bcd\u76f8\u7ed3\u5408\uff0c\u5e76\u5728\u7a0d\u540e\u7684\u72b6\u6001\u4e2d\u4f7f\u7528\u3002\u7136\u540e\uff0c\u6211\u4eec\u6839\u636e\u9700\u8981\u91cd\u590d\u6b64\u8fc7\u7a0b\uff0c\u76f4\u81f3\u8fbe\u5230\u6240\u9700\u7684\u65f6\u95f4\u6b65\u9aa4\u3002<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204210610.gif\" style=\"height:300px\">\n<\/p>\n<ul>\n<li>\n<p><a href=\"https:\/\/medium.com\/perceptronai\/recurrent-neural-network-an-introduction-for-beginners-1c13a541c906\">Image Source<\/a><\/p>\n<\/li>\n<li>\n<p>\u4ee4 $x$ \u8868\u793a\u8f93\u5165\u5c42\uff0c$h$ \u8868\u793a\u9690\u85cf\u5c42\uff0c$y$ \u8868\u793a\u8f93\u51fa\u5c42\u3002<\/p>\n<\/li>\n<li>\n<p>\u4ee4 $A, B \\text{ \u548c } C$ \u4e3a\u7528\u4e8e\u6539\u5584\u6a21\u578b\u8f93\u51fa\u7684\u4e00\u4e9b\u7f51\u7edc\u53c2\u6570\u3002<\/p>\n<\/li>\n<li>\n<p>\u5728\u4efb\u4f55\u7ed9\u5b9a\u65f6\u95f4 $t$\uff0c\u5f53\u524d\u8f93\u5165\u662f $x(t)$ \u548c $x(t-1)$ \u5904\u7684\u8f93\u5165\u7ec4\u5408\u3002<\/p>\n<p align=\"center\">\n<img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204246737.gif\" style=\"height:300px\">\n<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/www.simplilearn.com\/tutorials\/deep-learning-tutorial\/rnn\">Image Source<\/a><\/p>\n<p align=\"center\">\n<img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204334106.png\" style=\"height:300px\">\n<\/p>\n<\/li>\n<\/ul>\n<h4>The Hidden State of RNN Cells<\/h4>\n<hr \/>\n<ul>\n<li>\u5bf9\u4e8e\u8f93\u5165\u5e8f\u5217\u4e2d\u7684\u6bcf\u4e2a\u5143\u7d20\uff0c\u6bcf\u4e00\u5c42\u8ba1\u7b97\u4ee5\u4e0b\u51fd\u6570: $$ h<em>t = tanh\\left(W<\/em>{ih}x<em>t +b<\/em>{ih} + W<em>{hh}h<\/em>{(t-1)} + b_{hh}\\right), $$ \u5176\u4e2d $h_t$ \u662f\u65f6\u95f4 $t$ \u7684\u9690\u85cf\u72b6\u6001\uff0c$x<em>t$ \u662f\u65f6\u95f4 $t$ \u7684\u8f93\u5165\uff0c$h<\/em>{(t-1)}$ \u662f\u65f6\u95f4 $t-1$ \u65f6\u524d\u4e00\u5c42\u7684\u9690\u85cf\u72b6\u6001\u6216\u65f6\u95f4 0 \u65f6\u7684\u521d\u59cb\u9690\u85cf\u72b6\u6001\u3002<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204410404.gif\" style=\"height:300px\">\n<\/p>\n<ul>\n<li>Image by Michael Nguyen<\/li>\n<\/ul>\n<p>RNN\u7684\u4e24\u4e2a\u5e38\u89c1\u53d8\u4f53<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204503473.png\" style=\"height:400px\">\n<\/p>\n<h4><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/?size=100&id=91CnU00i6HLv&format=png&color=000000\" style=\"height:50px;display:inline\"> \u8fd9\u4e24\u4e2a\u53d8\u4f53\u7684\u8bbe\u8ba1\u52a8\u673a\u662f\uff1f<\/h4>\n<hr \/>\n<h3><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/dusk\/64\/000000\/memory-slot.png\" style=\"height:50px;display:inline\"> Long Term Short Memory (LSTM)<\/h3>\n<hr \/>\n<ul>\n<li>\n<p>\u5728\u53cd\u5411\u4f20\u64ad\u8fc7\u7a0b\u4e2d\uff0cRNN \u4f1a\u906d\u53d7\u68af\u5ea6\u6d88\u5931\u95ee\u9898\u7684\u5f71\u54cd\uff0c\u8fd9\u5b9e\u8d28\u4e0a\u4f1a\u4ea7\u751f<strong>\u77ed\u671f\u8bb0\u5fc6<\/strong>\u3002<\/p>\n<\/li>\n<li>\n<p>\u957f\u77ed\u671f\u8bb0\u5fc6 (LSTM) \u662f\u4e00\u79cd\u8bd5\u56fe\u4fdd\u5b58\u957f\u671f\u4fe1\u606f\u7684\u5faa\u73af\u5355\u5143\u3002<\/p>\n<\/li>\n<li>\n<p>LSTM \u5f15\u5165\u4e86\u4e00\u79cd\u4e0e\u9690\u85cf\u72b6\u6001\u5f62\u72b6\u76f8\u540c\u7684\u8bb0\u5fc6\u5355\u5143\uff0c\u65e8\u5728\u8bb0\u5f55\u66f4\u591a\u4fe1\u606f\u3002<\/p>\n<\/li>\n<li>\n<p>\u8bb0\u5fc6\u7531 3 \u4e2a\u4e3b\u8981\u95e8\u63a7\u5236\uff1a<\/p>\n<p><strong>\u8f93\u5165\u95e8<\/strong>\uff1a\u51b3\u5b9a\u4f55\u65f6\u5c06\u6570\u636e\u8bfb\u5165\u5355\u5143\u3002<\/p>\n<p><strong>\u8f93\u51fa\u95e8<\/strong>\uff1a\u8f93\u51fa\u5355\u5143\u4e2d\u7684\u6761\u76ee\u3002<\/p>\n<p><strong>\u9057\u5fd8\u95e8<\/strong>\uff1a\u4e00\u79cd\u91cd\u7f6e\u5355\u5143\u5185\u5bb9\u7684\u673a\u5236\u3002<\/p>\n<\/li>\n<li>\n<p>\u8fd9\u4e9b\u95e8\u4f1a\u4e86\u89e3\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u54ea\u4e9b\u4fe1\u606f\u4e0e\u9057\u5fd8\u6216\u8bb0\u4f4f\u6709\u5173\u3002\u8fd9\u4e9b\u95e8\u5305\u542b\u4e00\u4e2a S \u5f62\u6fc0\u6d3b\u51fd\u6570\uff08sigmoid\uff09\u3002<\/p>\n<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913204605991.png\" style=\"height:400px\">\n<\/p>\n<h4>Memory Cell<\/h4>\n<hr \/>\n<ul>\n<li>LSTM \u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65 $t$ \u4e0a\u63a5\u6536\u4e00\u4e2a\u8f93\u5165\u5411\u91cf $\\boldsymbol{x}_t$, \u66f4\u65b0\u5176\u7ec6\u80de\u72b6\u6001 $\\boldsymbol{C}_t$\u548c\u9690\u85cf\u72b6\u6001 $\\boldsymbol{h}_t$ \u3002\u5176\u4e2d, \u7ec6\u80de\u72b6\u6001 (cell state) \u662f LSTM \u7684\u65b0\u6982\u5ff5\uff0c\u6709\u65f6\u5019\u4e5f\u88ab\u79f0\u4e3a\u5019\u9009\u8bb0\u5fc6\u5355\u5143 $\\tilde{C}_t$ (candidate memory cell), \u5b83\u7c7b\u4f3c\u4e8e\u4f20\u9001\u5e26, \u662f LSTM \u7684\u5173\u952e, \u76f4\u63a5\u8d2f\u7a7f\u6574\u4e2a\u6a21\u578b, $C_t$\u5728\u4f20\u9012\u4e2d\u53ea\u6709\u4e00\u4e9b\u5c11\u91cf\u7684\u7ebf\u6027\u4ea4\u4e92, \u4f7f\u5f97\u4fe1\u606f\u5728\u4e0a\u9762\u4f20\u9012\u5f88\u5bb9\u6613\uff0c\u8fd9\u5bf9\u8fdc\u8ddd\u79bb\u7684\u4fe1\u606f\u66f4\u597d\u7684\u4f20\u9012\u5230\u5f53\u524d\u8ba1\u7b97\u65f6\u523b\u6709\u5f88\u5927\u5e2e\u52a9\u3002\u4e5f\u5c31\u662f\u957f\u77ed\u671f\u8bb0\u5fc6\u7f51\u7edc\u540d\u5b57\u4e2d\u5bf9\u201c\u957f\u671f\u201d\u6982\u5ff5\u7684\u4e00\u79cd\u5b9e\u73b0\u3002<\/li>\n<li>\u5177\u4f53\u6765\u8bf4\uff0c\u8f93\u5165\u95e8 $I_t$\u63a7\u5236\u6211\u4eec\u901a\u8fc7 $\\tilde{C}_t$\u8003\u8651\u591a\u5c11\u65b0\u6570\u636e\uff0c\u800c\u9057\u5fd8\u95e8  $F_t$ \u51b3\u5b9a\u6211\u4eec\u4fdd\u7559\u4e86\u591a\u5c11\u65e7\u8bb0\u5fc6\u5355\u5143\u5185\u5bb9 $C_{t-1}$\u3002\u7531\u6b64\u5f97\u51fa\uff1a  $$ C_t = F_t \\odot C_{t-1} + I_{t} \\odot \\tilde{C}_t$$<\/li>\n<li>$\\odot$ \u662f\u5143\u7d20\u4e58\u79ef\u8fd0\u7b97\u7b26 (Hadamard)\u3002<\/li>\n<li>\u5982\u679c\u9057\u5fd8\u95e8\u59cb\u7ec8\u8fd1\u4f3c\u4e3a 1\uff0c\u8f93\u5165\u95e8\u59cb\u7ec8\u8fd1\u4f3c\u4e3a 0\uff0c\u5219\u8fc7\u53bb\u7684\u8bb0\u5fc6\u5355\u5143 $C_{t-1}$ \u5c06\u968f\u65f6\u95f4\u4fdd\u5b58\u5e76\u4f20\u9012\u5230\u5f53\u524d\u65f6\u95f4\u6b65\u9aa4\u3002<\/li>\n<li>\u5f15\u5165\u8fd9\u79cd\u8bbe\u8ba1\u53ef\u4ee5\u7f13\u89e3\u6d88\u5931\u68af\u5ea6\u95ee\u9898\uff08memory cell\u8def\u5f84\u4e2d\u7ecf\u5386\u7684\u8ba1\u7b97\u6bd4hidden state\u5c11\u4e00\u6b21\u76f8\u4e58\uff09\uff0c\u5e76\u66f4\u597d\u5730\u6355\u83b7\u5e8f\u5217\u5185\u7684\u957f\u8ddd\u79bb\u4f9d\u8d56\u5173\u7cfb\u3002<\/li>\n<li>\u6700\u540e\uff0c\u65f6\u95f4 $t$ \u7684\u9690\u85cf\u72b6\u6001\uff1a $$ H_t = O_t \\odot \\text{tanh}(C_t)\u3002$$<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913205104920.gif\" style=\"height:300px\">\n<\/p>\n<ul>\n<li><a href=\"https:\/\/becominghuman.ai\/long-short-term-memory-part-1-3caca9889bbc\">Image Source<\/a><\/li>\n<\/ul>\n<p>\u5728 LSTM \u7684\u8ba1\u7b97\u8fc7\u7a0b\u4e2d, \u7b2c\u4e00\u6b65\u51b3\u5b9a\u6211\u4eec\u4f1a\u4ece\u4e0a\u4e00\u65f6\u523b\u7684\u9690\u85cf\u72b6\u6001\u4e2d\u4e1f\u5f03\uff08\u5fd8\u8bb0\uff09\u4ec0\u4e48\u4fe1\u606f\u3002\u5982\u56fe, \u5176\u4e2d $\\sigma$\u88ab\u79f0\u4f5c\u9057\u5fd8\u95e8 (forget gate), \u672c\u8d28\u4e0a\u662f\u4e00\u4e2a\u5e26\u6709 Sigmoid \u51fd\u6570\u7684\u795e\u7ecf\u7f51\u7edc\u5c42, \u901a\u8fc7\u8bfb\u53d6 $\\boldsymbol{h}_{t-1}$ \u548c $\\boldsymbol{x}_t$, \u8f93\u51fa\u4e00\u4e2a\u5728 0 \u5230 1 \u4e4b\u95f4\u7684\u6570\u503c\u7ed9\u5230\u7ec6\u80de\u72b6\u6001 $\\boldsymbol{C}_{t-1}$ \u3002\u901a\u8fc7\u76f8\u4e58\u7684\u64cd\u4f5c\u6765\u51b3\u5b9a\u4fdd\u7559\u591a\u5c11 $\\boldsymbol{C}_{t-1}$  \u4e2d\u7684\u4fe1\u606f, 1 \u8868\u793a\u4fdd\u7559\u5168\u90e8\u4fe1\u606f, 0 \u8868\u793a\u9057\u5fd8\u5168\u90e8\u4fe1\u606f\u3002\u516c\u5f0f\u8868\u793a\u5982\u4e0b<br \/>\n$$<br \/>\nf_t=\\sigma\\left(\\boldsymbol{W}_f *\\left[\\boldsymbol{h}_{t-1}, \\boldsymbol{x}_t\\right]+b_f\\right)<br \/>\n$$<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913205325161.png\" style=\"height:300px\">\n<\/p>\n<p>\u7b2c\u4e8c\u6b65\u51b3\u5b9a\u4f1a\u4ece\u5f53\u524d\u65f6\u523b\u7684\u8f93\u5165\u4fe1\u606f\u4e2d, \u9009\u62e9\u4ec0\u4e48\u4fe1\u606f\u8fdb\u884c\u5904\u7406, \u5982\u4e0b\u56fe\u6240\u793a\u3002\u5176\u4e2d $\\sigma$ \u88ab\u79f0\u4f5c\u8f93\u5165\u95e8 input $i_t$, \u516c\u5f0f\u8868\u793a\u5982\u4e0b:<br \/>\n$$<br \/>\n\\boldsymbol{i}_t=\\sigma\\left(\\boldsymbol{W}_i \\cdot\\left[\\boldsymbol{h}_{t-1}, \\boldsymbol{x}_t\\right]+b_i\\right)<br \/>\n$$<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913205510640.png\" style=\"height:300px\">\n<\/p>\n<p>\u7b2c\u4e09\u6b65, \u4f7f\u7528 $\\tanh$ \u6fc0\u6d3b\u51fd\u6570\u521b\u5efa\u4e00\u4e2a\u65b0\u7684\u5019\u9009\u7ec6\u80de\u72b6\u6001  $\\tilde{\\boldsymbol{C}}_t$, \u5982\u4e0b\u56fe\u6240\u793a\u3002\u672c\u8d28\u4e0a\u662f\u4e00\u4e2a\u5e26\u6709 $\\tanh$ \u51fd\u6570\u7684\u795e\u7ecf\u7f51\u7edc\u5c42, \u901a\u8fc7\u8bfb\u53d6 $\\boldsymbol{h}_{t-1}$ \u548c $\\boldsymbol{x}_t$ \uff0c\u8f93\u51fa $\\tilde{\\boldsymbol{C}}_t$, \u8fd9\u4e2a $\\tilde{\\boldsymbol{C}}_t$\u53ef\u4ee5\u4e00\u5b9a\u7a0b\u5ea6\u4e0a\u4ee3\u8868 $\\left[\\boldsymbol{h}_{t-1}, \\boldsymbol{x}_t\\right]$, \u5373\u4ee3\u8868\u8f93\u5165\u4fe1\u606f, \u516c\u5f0f\u8868\u793a\u5982\u4e0b:<br \/>\n$$<br \/>\n\\tilde{\\boldsymbol{C}}_t=\\tanh \\left(\\boldsymbol{W}_C \\cdot\\left[\\boldsymbol{h}_{t-1}, \\boldsymbol{x}_t\\right]+b_C\\right)<br \/>\n$$<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913205843441.png\" style=\"height:300px\">\n<\/p>\n<p>\u6700\u540e\u4e00\u6b65, \u51b3\u5b9a\u7740\u5f53\u524d\u65f6\u523b\u9690\u85cf\u72b6\u6001\u7684\u8f93\u51fa\u7ed3\u679c, \u5982\u4e0b\u56fe\u6240\u793a\u3002\u5176\u4e2d $\\sigma$ \u88ab\u79f0\u4f5c\u8f93\u51fa\u95e8 (forget gate), \u672c\u8d28\u4e0a\u8fd8\u662f\u4e00\u4e2a\u5e26\u6709 Sigmoid \u51fd\u6570\u7684\u795e\u7ecf\u7f51\u7edc\u5c42, \u5b83\u7684\u8ba1\u7b97\u7ed3\u679c\u51b3\u5b9a\u7740\u5f53\u524d\u65f6\u523b\u7684\u8f93\u51fa\u7ed3\u679c\u6709\u591a\u5c11\u6bd4\u4f8b\u80fd\u8fdb\u5165\u5230\u4e0b\u4e00\u65f6\u523b\u3002\u5177\u4f53\u6765\u8bf4, \u8f93\u51fa\u95e8\u7684\u8ba1\u7b97\u7ed3\u679c, \u5c06\u4e0e\u66f4\u65b0\u540e\u7684\u4e14\u7ecf\u8fc7\u6fc0\u6d3b\u51fd\u6570 $\\tanh$ \u6620\u5c04\u540e\u7684\u7ec6\u80de\u72b6\u6001  $\\boldsymbol{C}_t$ \u76f8\u4e58, \u4f5c\u4e3a\u5f53\u524d\u65f6\u523b\u9690\u85cf\u72b6\u6001 $\\boldsymbol{h}_t$ \u7684\u8f93\u51fa\u7ed3\u679c\u3002\u516c\u5f0f\u8868\u793a\u5982\u4e0b:<br \/>\n$$<br \/>\n\\begin{gathered}<br \/>\n\\boldsymbol{o}_t=\\sigma\\left(\\boldsymbol{W}_o *\\left[\\boldsymbol{h}_{t-1}, \\boldsymbol{x}_t\\right]+b_o\\right) \\\\<br \/>\n\\boldsymbol{h}_t=\\boldsymbol{o}_t \\odot \\tanh \\left(\\boldsymbol{C}_t\\right)<br \/>\n\\end{gathered}<br \/>\n$$<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913205843441.png\" style=\"height:300px\">\n<\/p>\n<pre><code class=\"language-python\">rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)  # batch_first=False\nx = torch.randn(5, 3, 10)  # 5 words per senetence, 3 sentences (batch_size), embedding dimension of each word is 10\nh0 = torch.randn(2, 3, 20)  # initialize hidden states per layer\nc0 = torch.randn(2, 3, 20)  # initialize memory per layer\noutput, (hn, cn) = rnn(x, (h0, c0))\nprint(f&#039;shapes: output - {output.shape}, hidden - {hn.shape}, memory - {cn.shape}&#039;)<\/code><\/pre>\n<pre><code>shapes: output - torch.Size([5, 3, 20]), hidden - torch.Size([2, 3, 20]), memory - torch.Size([2, 3, 20])<\/code><\/pre>\n<h3><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/color\/96\/000000\/front-gate-open.png\" style=\"height:50px;display:inline\"> Gated Recurrent Unit (GRU)<\/h3>\n<hr \/>\n<ul>\n<li>\u4e0e\u5e38\u89c4 RNN \u4e0d\u540c\uff0c\u95e8\u63a7\u5faa\u73af\u5355\u5143 (GRU) \u652f\u6301\u9690\u85cf\u72b6\u6001\u7684\u95e8\u63a7\u3002<\/li>\n<li>GRU \u6709\u4e24\u79cd\u673a\u5236\u6765\u63a7\u5236\u4f55\u65f6\u5e94\u66f4\u65b0\u9690\u85cf\u72b6\u6001\uff1a<\/li>\n<li><strong>\u91cd\u7f6e\u95e8<\/strong>\uff1a\u5141\u8bb8\u63a7\u5236\u5e94\u8bb0\u4f4f\u591a\u5c11\u5148\u524d\u72b6\u6001\uff0c\u6709\u52a9\u4e8e\u6355\u83b7\u5e8f\u5217\u4e2d\u7684\u77ed\u671f\u4f9d\u8d56\u5173\u7cfb\u3002<\/li>\n<li><strong>\u66f4\u65b0\u95e8<\/strong>\uff1a\u5141\u8bb8\u63a7\u5236\u65b0\u72b6\u6001\u4e2d\u6709\u591a\u5c11\u53ea\u662f\u65e7\u72b6\u6001\u7684\u526f\u672c\uff0c\u6709\u52a9\u4e8e\u6355\u83b7\u5e8f\u5217\u4e2d\u7684\u957f\u671f\u4f9d\u8d56\u5173\u7cfb\u3002<\/li>\n<li>\u4e0e LSTMS \u4e0d\u540c\uff0cGRU \u6ca1\u6709\u8bb0\u5fc6\u7ec4\u4ef6\uff0c\u56e0\u6b64\u66f4\u65b0\u901f\u5ea6\u66f4\u5feb\uff08\u4ece\u800c\u53ef\u4ee5\u52a0\u5feb\u8bad\u7ec3\u901f\u5ea6\uff09\uff0c\u4f46\u901a\u5e38 LSTM \u8868\u73b0\u66f4\u597d\u3002<\/li>\n<\/ul>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913210213976.png\" style=\"height:300px\">\n<\/p>\n<ul>\n<li>\u5047\u8bbe\u8f93\u5165\u662f\u7ed9\u5b9a\u65f6\u95f4\u6b65\u957f $t$ \u7684\u5c0f\u6279\u91cf $X_t \\in \\mathbb{R}^{n\\times d}$\uff0c\u524d\u4e00\u4e2a\u65f6\u95f4\u6b65\u957f\u7684\u9690\u85cf\u72b6\u6001\u662f $H_{t\u22121}\\in \\mathbb{R}^{n\\times h}$\u3002<\/li>\n<li>\u91cd\u7f6e\u95e8$R_t \\in \\mathbb{R}^{n \\times h}$ \u548c\u66f4\u65b0\u95e8$Z_t \\in \\mathbb{R}^{n \\times h}$\u8ba1\u7b97\u5982\u4e0b\uff1a $$ R_t = \\sigma(X_tW_{xr} + H_{t-1}W_{hr} +b_r), $$ $$ Z_t = \\sigma(X_tW_{xz} + H_{t-1}W_{hz} +b_z), $$ \u5176\u4e2d $W_{xr}, W_{xz} \\in \\mathbb{R}^{d \\times h}$  \u548c $W_{hr}, W_{hz} \\in \\mathbb{R}^{h \\times h}$ \u4e3a\u6743\u91cd\u53c2\u6570\uff0c$b_r, b_z \\in \\mathbb{R}^{1 \\times h}$\u5b58\u5728\u504f\u89c1\u3002<\/li>\n<\/ul>\n<h4>GRUs Hidden State<\/h4>\n<hr \/>\n<ul>\n<li>\u65f6\u95f4\u6b65 $t$ \u7684 <em>\u5019\u9009<\/em> \u9690\u85cf\u72b6\u6001 $\\tilde{H}_{t} \\in \\mathbb{R}^{n \\times h}$ \u5b9a\u4e49\u4e3a\uff1a $$ \\tilde{H}_{t} = \\text{tanh}\\left(X_t W_{xh} + (R_t \\odot H_{t-1})W_{hh} \\right) + b_h$$<\/li>\n<li>\u7ed3\u679c\u4e3a\u5019\u9009\uff0c\u56e0\u4e3a\u6211\u4eec\u4ecd\u9700\u7ed3\u5408 <em>\u66f4\u65b0\u95e8<\/em> \u7684\u4f5c\u7528\u3002<\/li>\n<li>\u6700\u540e\uff0c\u65b0\u7684\u9690\u85cf\u72b6\u6001 $H_t$ \u548c\u65f6\u95f4\u6b65 $t$ \u4e2d GRU \u7684\u6700\u7ec8\u66f4\u65b0\uff1a$$ H_t = Z_t \\odot H_{t-1} +(1-Z_t) \\odot \\tilde{H}_t\u3002 $$<\/li>\n<li>\u6bcf\u5f53\u66f4\u65b0\u95e8  $Z_t$  \u63a5\u8fd1 1 \u65f6\uff0c\u6211\u4eec\u53ea\u9700\u4fdd\u7559\u65e7\u72b6\u6001\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0c\u6765\u81ea $X_t$ \u7684\u4fe1\u606f\u57fa\u672c\u4e0a\u88ab\u5ffd\u7565\uff0c\u4ece\u800c\u6709\u6548\u5730\u8df3\u8fc7\u4e86\u4f9d\u8d56\u94fe\u4e2d\u7684\u65f6\u95f4\u6b65\u9aa4 $t$\u3002<\/li>\n<li>\u76f8\u53cd\uff0c\u6bcf\u5f53 $Z_t$ \u63a5\u8fd1 0 \u65f6\uff0c\u65b0\u7684\u6f5c\u5728\u72b6\u6001 $H_t$  \u5c31\u4f1a\u63a5\u8fd1\u5019\u9009\u6f5c\u5728\u72b6\u6001 $\\tilde{H}_t$\u3002<\/li>\n<li>\u8fd9\u4e9b\u8bbe\u8ba1\u53ef\u4ee5\u5e2e\u52a9\u6211\u4eec\u5e94\u5bf9 RNN \u4e2d\u7684\u68af\u5ea6\u6d88\u5931\u95ee\u9898\uff0c\u5e76\u66f4\u597d\u5730\u6355\u83b7\u5177\u6709\u8f83\u5927\u65f6\u95f4\u6b65\u957f\u8ddd\u79bb\u7684\u5e8f\u5217\u7684\u4f9d\u8d56\u5173\u7cfb\u3002<\/li>\n<\/ul>\n<pre><code class=\"language-python\">rnn = nn.GRU(input_size=10, hidden_size=20, num_layers=2)  \nx = torch.randn(5, 3, 10)  # 5 words per senetence, 3 sentences (batch_size), embedding dimension of each word is 10\nh0 = torch.randn(2, 3, 20)  # initialize hidden states per layer\noutput, hn = rnn(x, h0)\nprint(f&#039;shapes: output - {output.shape}, hidden - {hn.shape}&#039;)<\/code><\/pre>\n<pre><code>shapes: output - torch.Size([5, 3, 20]), hidden - torch.Size([2, 3, 20])<\/code><\/pre>\n<h3><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/bubbles\/50\/000000\/fire-element.png\" style=\"height:50px;display:inline\"> PyTorch RNN Model Example<\/h3>\n<hr \/>\n<p>Following is an example of building a classifier with LSTMs.<\/p>\n<pre><code class=\"language-python\">#\u73af\u5883\u914d\u7f6e\n# -*- coding: utf-8 -*-\nimport numpy as np\nimport pickle as pkl\nfrom tqdm import tqdm\nfrom torch.utils.data import Dataset, DataLoader\nimport torch.nn as nn\nimport time\nimport torch\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\n\n# \u8d85\u53c2\u6570\u8bbe\u7f6e\ndata_path =  &#039;.\/datasets\/rnndata\/data.txt&#039;              \nvocab_path = &#039;.\/datasets\/rnndata\/vocab.pkl&#039;             \nsave_path = &#039;.\/datasets\/rnndata\/rnn.ckpt&#039;                    \nembedding_pretrained = torch.tensor(np.load(&#039;.\/datasets\/rnndata\/embedding_Tencent.npz&#039;)[&quot;embeddings&quot;].astype(&#039;float32&#039;)) \nembed = embedding_pretrained.size(1)        \ndropout = 0.5                              \nnum_classes = 2                             \nnum_epochs = 50                            \nbatch_size = 128                             \npad_size = 50                                \nlearning_rate = 1e-3                        \nhidden_size = 128                            \nnum_layers = 2                               \nMAX_VOCAB_SIZE = 10000                        <\/code><\/pre>\n<pre><code class=\"language-python\">import pickle\nimport numpy as np\n\n# \u52a0\u8f7d\u8bcd\u6c47\u8868\nwith open(&#039;.\/datasets\/rnndata\/\/vocab.pkl&#039;, &#039;rb&#039;) as f:\n    vocab = pickle.load(f)\n\n# \u52a0\u8f7d\u9884\u8bad\u7ec3\u8bcd\u5411\u91cf\nembeddings = np.load(&#039;.\/datasets\/rnndata\/embedding_Tencent.npz&#039;)[&#039;embeddings&#039;]\n\n# \u793a\u4f8b\uff1a\u83b7\u53d6\u4e00\u4e2a\u8bcd\u7684\u5411\u91cf\u8868\u793a\nword = &#039;\u6211&#039;\nif word in vocab:\n    word_index = vocab[word]\n    word_vector = embeddings[word_index]\n    print(f&quot;\u8bcd &#039;{word}&#039; \u7684\u5411\u91cf\u8868\u793a\u662f\uff1a&quot;, word_vector)\n    print(word_vector.shape)\nelse:\n    print(f&quot;\u8bcd &#039;{word}&#039; \u4e0d\u5728\u8bcd\u6c47\u8868\u4e2d&quot;)\n<\/code><\/pre>\n<pre><code>\u8bcd '\u6211' \u7684\u5411\u91cf\u8868\u793a\u662f\uff1a [ 0.25011599 -0.36695799  0.065014    0.010725    0.231398   -0.177817\n  0.064359   -0.005259    0.115888    0.154       0.17193501  0.07247\n -0.003175   -0.09248     0.20276    -0.030792   -0.30699101 -0.289693\n -0.055264   -0.189153    0.122888    0.081699    0.017909    0.15846901\n  0.14746401  0.079238   -0.224966    0.14583699  0.182973   -0.149864\n -0.15604401  0.044855   -0.237059    0.174146   -0.108293   -0.066462\n  0.140773    0.092687   -0.124868   -0.026098    0.167881   -0.117048\n  0.39074999 -0.036812   -0.051702   -0.161367   -0.355791   -0.311515\n -0.090306    0.084679   -0.18447199  0.090339   -0.098312    0.25659499\n  0.29255399  0.27464801  0.039325    0.150774    0.239049   -0.011787\n -0.014104    0.13506199  0.151537   -0.208729   -0.171538    0.08003\n -0.116087    0.159768   -0.061878   -0.149166   -0.065586    0.029528\n -0.020271    0.098718   -0.068513    0.238489    0.174631   -0.003655\n  0.161002   -0.002617    0.202535    0.276254    0.03933     0.008368\n  0.258403   -0.25468501 -0.046402    0.415975   -0.36980301 -0.070436\n  0.165418    0.167119    0.047878    0.105566   -0.127156    0.189587\n -0.022974   -0.21011201 -0.29733899  0.071683   -0.111609    0.26536801\n -0.108999    0.24790099 -0.43430701  0.216444    0.033944   -0.55144203\n -0.072594    0.00983    -0.028967   -0.181977   -0.032623   -0.179671\n  0.128933   -0.087196    0.12034    -0.221466   -0.20084099 -0.21887299\n  0.277069   -0.36693299  0.003879   -0.027925    0.069605   -0.057702\n -0.010946   -0.035958    0.039027    0.36728701  0.044901    0.128941\n -0.039109   -0.088411   -0.21058699 -0.195746   -0.26508701  0.117322\n  0.057173   -0.134257   -0.214258   -0.247163   -0.27433899  0.054974\n -0.111022   -0.567581    0.212942   -0.222993   -0.027288    0.198594\n -0.052249    0.149065   -0.24225099 -0.19462501 -0.040973   -0.00749\n  0.168107   -0.079271    0.26327199 -0.370942    0.190593    0.015323\n  0.115155    0.074836    0.307684    0.20419499 -0.105276    0.29840299\n  0.151988    0.045363   -0.12802801 -0.065257   -0.150151    0.13884\n -0.07511     0.019008   -0.181566   -0.044517    0.208922    0.067851\n  0.059112   -0.029587   -0.138191   -0.009103   -0.083886   -0.25208199\n -0.224802    0.45078599  0.100208   -0.050416    0.050791    0.095502\n -0.180603    0.003119    0.17693201  0.36746699 -0.042056   -0.33071601\n -0.165911    0.19402801]\n(200,)<\/code><\/pre>\n<pre><code class=\"language-python\"># dataset load\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom collections import Counter\n\ndata_path = &#039;.\/datasets\/rnndata\/data.txt&#039;\n\ndef load_raw_data(path):\n    data = []\n    with open(path, &#039;r&#039;, encoding=&#039;gbk&#039;) as f:\n        for line in f:\n            lin = line.strip()\n            if not lin:\n                continue\n            label, content = lin.split(&#039;   ####    &#039;)\n            data.append((content, int(label)))\n    return data\n\nraw_data = load_raw_data(data_path)\n<\/code><\/pre>\n<pre><code class=\"language-python\">def data_summary(data):\n    print(f&quot;\u6570\u636e\u96c6\u4e2d\u6837\u672c\u6570\u91cf: {len(data)}&quot;)\n    lengths = [len(sample[0]) for sample in data]\n    print(f&quot;\u6837\u672c\u7684\u5e73\u5747\u957f\u5ea6: {np.mean(lengths):.2f}&quot;)\n    print(f&quot;\u6837\u672c\u7684\u6700\u5927\u957f\u5ea6: {np.max(lengths)}&quot;)\n    print(f&quot;\u6837\u672c\u7684\u6700\u5c0f\u957f\u5ea6: {np.min(lengths)}&quot;)\n\ndata_summary(raw_data)\n<\/code><\/pre>\n<pre><code>\u6570\u636e\u96c6\u4e2d\u6837\u672c\u6570\u91cf: 119988\n\u6837\u672c\u7684\u5e73\u5747\u957f\u5ea6: 66.05\n\u6837\u672c\u7684\u6700\u5927\u957f\u5ea6: 260\n\u6837\u672c\u7684\u6700\u5c0f\u957f\u5ea6: 3<\/code><\/pre>\n<pre><code class=\"language-python\">for i in range(10):\n    print(raw_data[i])<\/code><\/pre>\n<pre><code>('?\u66f4\u535a\u4e86\uff0c\u7206\u7167\u4e86\uff0c\u5e05\u7684\u5440\uff0c\u5c31\u662f\u8d8a\u6765\u8d8a\u7231\u4f60\uff01\u751f\u5feb\u50bb\u7f3a[\u7231\u4f60][\u7231\u4f60][\u7231\u4f60]', 1)\n('@\u5f20\u6653\u9e4fjonathan \u571f\u8033\u5176\u7684\u4e8b\u8981\u8ba4\u771f\u5bf9\u5f85[\u54c8\u54c8]\uff0c\u5426\u5219\u76f4\u63a5\u5f00\u9664\u3002@\u4e01\u4e01\u770b\u4e16\u754c \u5f88\u662f\u7ec6\u5fc3\uff0c\u9152\u5e97\u90fd\u5168\u90e8OK\u5566\u3002', 1)\n('\u59d1\u5a18\u90fd\u7fa1\u6155\u4f60\u5462\u2026\u8fd8\u6709\u62db\u8d22\u732b\u9ad8\u5174\u2026\u2026\/\/@\u7231\u5728\u8513\u5ef6-JC:[\u54c8\u54c8]\u5c0f\u5b66\u5f92\u4e00\u679a\uff0c\u7b49\u7740\u660e\u5929\u89c1\u60a8\u5462\/\/@\u674e\u6b23\u82b8SharonLee:\u5927\u4f6c\u8303\u513f[\u4e66\u5446\u5b50]', 1)\n('\u7f8e~~~~~[\u7231\u4f60]', 1)\n('\u68a6\u60f3\u6709\u591a\u5927\uff0c\u821e\u53f0\u5c31\u6709\u591a\u5927![\u9f13\u638c]', 1)\n('[\u82b1\u5fc3][\u9f13\u638c]\/\/@\u5c0f\u61d2\u732bMelody2011: [\u6625\u6696\u82b1\u5f00]', 1)\n('\u67d0\u95ee\u7b54\u793e\u533a\u4e0a\u6536\u5230\u4e00\u5927\u5b66\u751f\u53d1\u7ed9\u6211\u7684\u79c1\u4fe1\uff1a\u201c\u5076\u559c\u6b22\u963f\u59e8\uff01\u5076\u662f\u963f\u59e8\u63a7\uff01\u201d\u6211\u56de\u4ed6\uff1a\u201c\u963f\u59e8\u7a00\u996d\u5c0f\u76c6\u53cb\uff01\u5076\u662f\u5c0f\u76c6\u53cb\u63a7\uff01\u201d [\u54c8\u54c8]', 1)\n('\u5403\u8d27\u4eec\u65e0\u4e0d\u5567\u5567\u79f0\u5947\uff0c\u597d\u4e0d\u559c\u6b22\uff01PS:\u5199\u9519\u4e00\u4e2a\u5b57\uff01[\u54c8\u54c8]@\u68ee\u6797\u5c0f\u5929\u4f7f-\u6ce2\u742a @SEVEN\u53a6\u95e8\u6444\u5f71\u5e08 @\u65e5\u6708\u661f\u8fb0-\u5fc3\u5728\u8def\u4e0a @\u6bcf\u79cd\u578b\u53f7\u751f\u4e24\u80ce @\u5fd7\u8fdc\u5929\u4e0b\u884c @\u76d1\u63a7\u9632\u76d7\u5b89\u88c5XM @\u521b\u610f\u7f8e\u98dfsimon\u54e5 @\u6f2b\u6e38\u8005-\u5f3a\u5b50 @\u9648\u5c0fkitty\u732b@\u6e38\u5b50\u7684\u6b4c@solo\u5728\u53a6\u95e8', 1)\n('#Sweet Morning#From now on,love yourself,enjoy living then smile.\u4ece\u73b0\u5728\u5f00\u59cb\uff0c\u7231\u81ea\u5df1\uff0c\u4eab\u53d7\u751f\u6d3b\u5e76\u4e14\u5fae\u7b11\u3002[\u5475\u5475] [\u563b\u563b] [\u54c8\u54c8] [\u6324\u773c] [\u592a\u5f00\u5fc3] \u65e9\u5b89\u3001\u751c\u5fc3\u4eec', 1)\n('\u3010\u970d\u601d\u71d5\u5256\u8179\u4ea7\u4e0b\u201c\u5c0f\u6c5f\u6c5f\u201d \u8001\u516c\u843d\u6cea\u3011\u4eca\u66689\u65f6\u970d\u601d\u71d5\u4ea7\u4e0b\u4e00\u540d\u7537\u5a74\uff0c\u5b9d\u5b9d\u91cd8\u65a43\u4e24\uff0c\u6bcd\u5b50\u5e73\u5b89\u3002\u675c\u6c5f\u7684\u8138\u4e0a\u6d0b\u6ea2\u7740\u505a\u7238\u7238\u7684\u6b23\u559c\uff1a\u5b9d\u5b9d\u5c0f\u540d\u53eb\u201c\u5c0f\u6c5f\u6c5f\u201d\uff0c\u773c\u775b\u50cf\u4ed6\uff0c\u9f3b\u5b50\u548c\u5634\u5df4\u5219\u50cf\u970d\u601d\u71d5\uff0c\u770b\u5230\u5b9d\u8d1d\u5c31\u5fcd\u4e0d\u4f4f\u843d\u6cea\uff01\u606d\u559c@\u675c\u6c5f\u4fa6\u5bdf\u8bb0 @\u970d\u601d\u71d5 \uff0c\u795d\u798f\u201c\u5c0f\u6c5f\u6c5f\u201d\u5728\u7231\u91cc\u5065\u5eb7\u5730\u6210\u957f[\u7231\u4f60]...http:\/\/t.cn\/z8EwSPU', 1)<\/code><\/pre>\n<pre><code class=\"language-python\"># \u6587\u672c\u957f\u5ea6\u5206\u5e03\ndef plot_length_distribution(data, title):\n    lengths = [len(sample[0]) for sample in data]\n    plt.figure(figsize=(10, 6))\n    sns.histplot(lengths, bins=50, kde=True)\n    plt.title(title)\n    plt.xlabel(&quot;Text length&quot;)\n    plt.ylabel(&quot;Frequency&quot;)\n    plt.show()\n\nplot_length_distribution(raw_data, &quot;Text length distribution&quot;)\n<\/code><\/pre>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913212729782.png\" style=\"height:300px\">\n<\/p>\n<pre><code class=\"language-python\">import re\n# \u5b9a\u4e49\u5e38\u91cf\nPAD = &#039;&lt;PAD&gt;&#039;\nUNK = &#039;&lt;UNK&gt;&#039;\nUSER = &#039;&lt;USER&gt;&#039;\nURL = &#039;&lt;URL&gt;&#039;\nEMOJI = &#039;&lt;EMOJI&gt;&#039;\n\n# \u8868\u60c5\u7b26\u53f7\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\nemoji_pattern = re.compile(r&#039;\\[.*?\\]&#039;)\n\ndef preprocess_text(text):\n    # \u53bb\u9664URL\n    text = re.sub(r&#039;http[s]?:\/\/\\S+&#039;, URL, text)\n    # \u66ff\u6362@\u7528\u6237\n    text = re.sub(r&#039;@\\S+&#039;, USER, text)\n    # \u66ff\u6362\u8868\u60c5\u7b26\u53f7\n    text = emoji_pattern.sub(EMOJI, text)\n    # \u53bb\u9664\u6807\u70b9\u7b26\u53f7\n    text = re.sub(r&#039;[^\\w\\s]&#039;, &#039;&#039;, text)\n    return text<\/code><\/pre>\n<pre><code class=\"language-python\">#\u6570\u636e\u9884\u5904\u7406\ndef load_dataset(path, pad_size, tokenizer, vocab):\n    contents = [] \n    n=0\n    with open(path, &#039;r&#039;, encoding=&#039;gbk&#039;) as f: \n        for line in tqdm(f):\n            lin = line.strip() \n            if not lin: \n                continue\n            label,content = lin.split(&#039;    ####    &#039;) \n            content = preprocess_text(content)   \n            words_line = [] \n            token = tokenizer(content) \n            # print(token)\n            seq_len = len(token) \n            if pad_size:\n\n                if seq_len &lt; pad_size:\n                    token.extend([vocab.get(PAD)] * (pad_size - len(token))) \n                else:\n                    token = token[:pad_size] \n                    seq_len = pad_size\n\n            for word in token:\n                words_line.append(vocab.get(word, vocab.get(UNK)))\n            n+=1\n            contents.append((words_line, int(label))) \n    train, X_t = train_test_split(contents, test_size=0.2, random_state=42) \n    dev,test= train_test_split(X_t, test_size=0.5, random_state=42)  \n    return train,dev,test\n\ndef get_data(): \n    tokenizer = lambda x: [y for y in x]  #jieba\n    vocab = pkl.load(open(vocab_path, &#039;rb&#039;))  \n    # print(&#039;tokenizer&#039;,tokenizer)\n    print(&#039;vocab&#039;,vocab)\n    print(f&quot;Vocab size: {len(vocab)}&quot;) \n\n    train,dev,test = load_dataset(data_path, pad_size, tokenizer, vocab) \n    return vocab, train, dev, test<\/code><\/pre>\n<pre><code class=\"language-python\">#\u5b9a\u4e49\u6570\u636e\u96c6\u7c7b\uff0c\u7528\u4e8e\u5904\u7406\u6587\u672c\u6570\u636e\u3002\nclass TextDataset(Dataset):\n    def __init__(self, data):\n        self.device = torch.device(&#039;cuda&#039;) if torch.cuda.is_available() else torch.device(&#039;cpu&#039;) \n        self.x = torch.LongTensor([x[0] for x in data]).to(self.device) \n        self.y = torch.LongTensor([x[1] for x in data]).to(self.device) \n    def __getitem__(self,index): \n        self.text = self.x[index] \n        self.label = self.y[index] \n        return self.text, self.label \n    def __len__(self):\n        return len(self.x) <\/code><\/pre>\n<pre><code class=\"language-python\"># \u5b9a\u4e49 \u6a21\u578b\nclass Model(nn.Module):\n    def __init__(self):\n        super(Model, self).__init__()\n\n        self.embedding = nn.Embedding.from_pretrained(embedding_pretrained, freeze=False)\n\n        self.rnn = nn.RNN(embed, hidden_size, num_layers, bidirectional=True, batch_first=True, dropout=dropout)\n\n        self.fc = nn.Linear(hidden_size * 2, num_classes)\n\n    def forward(self, x): \n        out = self.embedding(x) \n        out, _ = self.rnn(out) \n        out = self.fc(out[:, -1, :]) #RNN\u7684\u8f93\u51fa out \u5f62\u72b6\u4e3a (batch_size, seq_length, hidden_size * 2)\n        return out<\/code><\/pre>\n<pre><code class=\"language-python\"># \u6743\u91cd\u521d\u59cb\u5316\uff0c\u9ed8\u8ba4xavier\ndef init_network(model, method=&#039;xavier&#039;, exclude=&#039;embedding&#039;):\n    for name, w in model.named_parameters():\n        if exclude not in name:\n            if &#039;weight&#039; in name:\n                if method == &#039;xavier&#039;:\n                    nn.init.xavier_normal_(w)\n                elif method == &#039;kaiming&#039;:\n                    nn.init.kaiming_normal_(w)\n                else:\n                    nn.init.normal_(w)\n            elif &#039;bias&#039; in name:\n                nn.init.constant_(w, 0) \n            else:\n                pass<\/code><\/pre>\n<pre><code class=\"language-python\">#\u635f\u5931\u7ed8\u5236\ndef plot_loss(train_loss):\n    plt.figure(figsize=(10, 7)) \n    x = list(range(len(train_loss))) \n    plt.plot(x, train_loss, alpha=0.9, linewidth=2, label=&#039;train acc&#039;) \n    plt.xlabel(&#039;Epoch&#039;) \n    plt.ylabel(&#039;loss&#039;) \n    plt.legend(loc=&#039;best&#039;)  \n#\u51c6\u786e\u7387\u7ed8\u5236   \ndef plot_acc(train_acc):\n    plt.figure(figsize=(10, 7))\n    x = list(range(len(train_acc))) \n    plt.plot(x, train_acc, alpha=0.9, linewidth=2, label=&#039;train acc&#039;) \n    plt.xlabel(&#039;Epoch&#039;)\n    plt.ylabel(&#039;Acc&#039;)\n    plt.legend(loc=&#039;best&#039;) <\/code><\/pre>\n<pre><code class=\"language-python\">from sklearn.metrics import accuracy_score\n#\u5b9a\u4e49\u6a21\u578b\u8bad\u7ec3\ndef train( model, dataloaders):\n    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) \n    loss_function = torch.nn.CrossEntropyLoss() \n\n    dev_best_loss = float(&#039;inf&#039;)  \n\n    device = torch.device(&#039;cuda&#039;) if torch.cuda.is_available() else torch.device(&#039;cpu&#039;) \n    print(&quot;Start Training...\\n&quot;)\n\n    plot_train_acc = []\n    plot_train_loss = []\n\n    for i in range(num_epochs):\n        step = 0  \n        train_lossi=0 \n        train_acci = 0 \n        for inputs, labels in dataloaders[&#039;train&#039;]: \n            model.train() \n\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n            optimizer.zero_grad()  \n            outputs = model(inputs)  \n            loss = loss_function(outputs, labels) \n            loss.backward() \n            optimizer.step()\n\n            step += 1  \n            true = labels.data.cpu() \n            predic = torch.max(outputs.data, 1)[1].cpu() \n\n            train_lossi += loss.item()\n            train_acci += metrics.accuracy_score(true, predic)\n        #\u5bf9\u9a8c\u8bc1\u96c6\u8fdb\u884c\u8bc4\u4f30    \n        dev_acc, dev_loss = dev_eval(model, dataloaders[&#039;dev&#039;], loss_function,Result_test=False)\n        if dev_loss &lt; dev_best_loss:\n            dev_best_loss = dev_loss\n            torch.save(model.state_dict(), save_path)\n\n        train_acc = train_acci\/step\n        train_loss = train_lossi\/step\n\n        plot_train_acc.append(train_acc)\n        plot_train_loss.append(train_loss)\n\n        print(&quot;epoch = {} :  train_loss = {:.3f}, train_acc = {:.2%}, dev_loss = {:.3f}, dev_acc = {:.2%}&quot;.\n                  format(i+1, train_loss, train_acc, dev_loss, dev_acc))\n\n    plot_loss(plot_train_loss)\n    plot_acc(plot_train_acc)\n    # \u52a0\u8f7d\u6700\u4f73\u6a21\u578b\u53c2\u6570\u5e76\u5728\u6d4b\u8bd5\u96c6\u4e0a\u8fdb\u884c\u8bc4\u4f30\n    model.load_state_dict(torch.load(save_path))#\u6a21\u578b\u52a0\u8f7d\n    model.eval()#\u8bbe\u7f6e\u9a8c\u8bc1\u6a21\u578b\n    test_acc, test_loss = dev_eval(model, dataloaders[&#039;test&#039;], loss_function,Result_test=True)#\u8f93\u51fa\u51c6\u786e\u7387\u548c\u635f\u5931\n    print(&#039;================&#039;*8)\n    print(&#039;test_loss: {:.3f}      test_acc: {:.2%}&#039;.format(test_loss, test_acc))\n\n# \u5bf9\u6d4b\u8bd5\u7ed3\u679c\u8fdb\u884c\u5206\u6790\ndef result_test(real, pred):\n    acc = accuracy_score(real, pred)#\u51c6\u786e\u7387\u8ba1\u7b97\n\n# \u6a21\u578b\u8bc4\u4f30\u51fd\u6570\u5b9a\u4e49\ndef dev_eval(model, data, loss_function,Result_test=False):\n\n    model.eval() \n    loss_total = 0  \n    predict_all = np.array([], dtype=int) \n    labels_all = np.array([], dtype=int) \n    with torch.no_grad():\n        # \u904d\u5386\u6570\u636e\u96c6\n        for texts, labels in data:\n            outputs = model(texts)  \n            loss = loss_function(outputs, labels) \n            loss_total += loss.item()\n            labels = labels.data.cpu().numpy() \n            predic = torch.max(outputs.data, 1)[1].cpu().numpy() \n\n            labels_all = np.append(labels_all, labels)\n            predict_all = np.append(predict_all, predic)\n    # \u8ba1\u7b97\u51c6\u786e\u7387\n    acc = metrics.accuracy_score(labels_all, predict_all)\n    # \u5982\u679c\u9700\u8981\u8fdb\u884c\u6d4b\u8bd5\u7ed3\u679c\u5206\u6790\uff0c\u5219\u8c03\u7528result_test\u51fd\u6570\n    if Result_test:\n        result_test(labels_all, predict_all)\n    else:\n        pass\n     # \u8fd4\u56de\u51c6\u786e\u7387\u548c\u5e73\u5747\u635f\u5931\n    return acc, loss_total \/ len(data)<\/code><\/pre>\n<pre><code class=\"language-python\"># \u8bbe\u7f6e\u968f\u673a\u79cd\u5b50\uff0c\u4ee5\u786e\u4fdd\u7ed3\u679c\u7684\u53ef\u91cd\u590d\u6027\nnp.random.seed(1)\ntorch.manual_seed(1)\ntorch.cuda.manual_seed_all(1)\ntorch.backends.cudnn.deterministic = True  # \u4fdd\u8bc1\u6bcf\u6b21\u7ed3\u679c\u4e00\u6837\n\nstart_time = time.time()\nprint(&quot;Loading data...&quot;)\nvocab, train_data, dev_data, test_data = get_data()# \u52a0\u8f7d\u6570\u636e\u96c6\n# \u521b\u5efa\u6570\u636e\u52a0\u8f7d\u5668\ndataloaders = {\n        &#039;train&#039;: DataLoader(TextDataset(train_data), batch_size, shuffle=True),\n        &#039;dev&#039;: DataLoader(TextDataset(dev_data), batch_size, shuffle=True),\n        &#039;test&#039;: DataLoader(TextDataset(test_data), batch_size, shuffle=True)\n}\nend_time = time.time()\ntime_dif=end_time - start_time\nprint(&quot;Time usage:&quot;, time_dif)#\u6570\u636e\u5904\u7406\u7528\u65f6\ndevice = torch.device(&#039;cuda&#039;) if torch.cuda.is_available() else torch.device(&#039;cpu&#039;)#\u8bbe\u5907\u9009\u62e9\nmodel = Model().to(device)\ninit_network(model)# \u521d\u59cb\u5316\u6a21\u578b\u53c2\u6570\ntrain(model, dataloaders)#\u8fdb\u884c\u6a21\u578b\u8bad\u7ec3\u53ca\u9a8c\u8bc1\u548c\u8bc4\u4f30\nprint(&#039;end&#039;)<\/code><\/pre>\n<pre><code>Loading data...\nvocab {' ': 0, '0': 1, '1': 2, '2': 3, '\uff1a': 4, '\u5927': 5, '\u56fd': 6, '\u56fe': 7, '(': 8, ')': 9, '3': 10, '\u4eba': 11, '\u5e74': 12, '5': 13, '\u4e2d': 14, '\u65b0': 15, '9': 16, '\u751f': 17, '\u91d1': 18, '\u9ad8': 19, '\u300a': 20, '\u300b': 21, '4': 22, '\u4e0a': 23, '8': 24, '\u4e0d': 25, '\u8003': 26, '\u4e00': 27, '6': 28, '\u65e5': 29, '\u5143': 30, '\u5f00': 31, '\u7f8e': 32, '\u4ef7': 33, '\u53d1': 34, '\u5b66': 35, '\u516c': 36, '\u6210': 37, '\u6708': 38, '\u5c06': 39, '\u4e07': 40, '7': 41, '\u57fa': 42, '\u5e02': 43, '\u51fa': 44, '\u5b50': 45, '\u884c': 46, '\u673a': 47, '\u4e1a': 48, '\u88ab': 49, '\u5bb6': 50, '\u80a1': 51, '\u7684': 52, '\u5728': 53, '\u7f51': 54, '\u5973': 55, '\u671f': 56, '\u5e73': 57, '\u623f': 58, '\u540d': 59, '\u4e09': 60, '-': 61, '\u4f1a': 62, '\u5730': 63, '\u573a': 64, '\u5168': 65, '\u5c0f': 66, '\u73b0': 67, '\u6709': 68, '\u5206': 69, '\u540e': 70, '\u79f0': 71, '\u7ec4': 72, '\u4e3a': 73, '\u4e0b': 74, '\u76d8': 75, '\u6700': 76, '\u201c': 77, '\u201d': 78, '\u624b': 79, '\u5929': 80, '\u672c': 81, '\u5229': 82, '\u9996': 83, '\u6218': 84, '\u957f': 85, '\u6e38': 86, '\u6d77': 87, '\u4e3b': 88, '\u8d77': 89, '\u52a8': 90, '\u5317': 91, '\u8d44': 92, '\u552e': 93, '\u80fd': 94, '\u91cd': 95, '\u65f6': 96, '\u7537': 97, '\u529b': 98, '\u5c45': 99, '\u62a5': 100, '\u70b9': 101, '\u81ea': 102, '\u5e03': 103, '.': 104, '\u63a8': 105, '\u524d': 106, 'C': 107, '\u4ea7': 108, '\u8d5b': 109, '\u662f': 110, 'P': 111, '\u6307': 112, '\u4e0e': 113, '\u4eac': 114, '\u591a': 115, '\u65b9': 116, '\u5546': 117, 'S': 118, 'A': 119, '\u7403': 120, '\u8f66': 121, '\u7406': 122, '\u82f1': 123, '\u5165': 124, '\u534e': 125, '\u53ef': 126, '\u5bf9': 127, '\u8054': 128, '\u5185': 129, '\u90e8': 130, '\u65af': 131, '\u897f': 132, '\u6587': 133, '\u52a0': 134, '\u4e24': 135, '\u660e': 136, '\u7535': 137, '\u54c1': 138, '\u5de5': 139, '\u4f5c': 140, '\u5ea6': 141, '\u6b7b': 142, '\u60c5': 143, '\u6536': 144, '\u62db': 145, '\u56de': 146, 'i': 147, '\u901a': 148, '\u57ce': 149, '\u4ea4': 150, '\u6cd5': 151, '\u4e1c': 152, '\u5f3a': 153, '\u4e4b': 154, '\u5468': 155, '\u6c11': 156, '\u6570': 157, '\u7528': 158, '\u6253': 159, '\u4ebf': 160, '\u79d1': 161, '\u7ecf': 162, '%': 163, '\u7ebf': 164, '\u5408': 165, '\u7b2c': 166, '\u9a6c': 167, 'O': 168, '\u5c71': 169, '\u8d85': 170, 'o': 171, '\u4e16': 172, '\u4f53': 173, '\u8c03': 174, '\u4e13': 175, '\u5458': 176, 'e': 177, '\u65e0': 178, '\u4e2a': 179, '\u8981': 180, '\u603b': 181, '\u7ea7': 182, '\u5b9a': 183, '\u4f20': 184, '\u5916': 185, 'D': 186, '\u8001': 187, '\u5efa': 188, '\u4fe1': 189, '\u8bd5': 190, '\u519b': 191, '\u906d': 192, '\u70ed': 193, '\u738b': 194, '\u518d': 195, '\u661f': 196, '\u6301': 197, '\u6295': 198, '\u83b7': 199, '\u8fdb': 200, '\u4e9a': 201, '\u5fc3': 202, '\u56db': 203, '\u5df4': 204, '\u6bd4': 205, 'I': 206, '\u95e8': 207, '\u8eab': 208, '\u6da8': 209, '\u620f': 210, '\u5b89': 211, '\u4fdd': 212, '\u5149': 213, '\u53d7': 214, '\u5c14': 215, '\u7814': 216, '\u8bc4': 217, '\u8ba1': 218, '\u6765': 219, '\u53f0': 220, '\u7eed': 221, 'a': 222, '\u4e8b': 223, '\u7279': 224, '\u9884': 225, '\u9762': 226, '\u653f': 227, '\u6821': 228, '\u8005': 229, 'T': 230, 'n': 231, '\u5173': 232, '\u589e': 233, '\u53f8': 234, '\u7cbe': 235, '\u5357': 236, '\u4e8c': 237, '\u8d2d': 238, '\u9009': 239, '\u706b': 240, '\u88c5': 241, '\u6e2f': 242, '\u4f4e': 243, '\u961f': 244, '\u5347': 245, '\u89c6': 246, '\u8dcc': 247, 'G': 248, '\u8bc1': 249, 'L': 250, '\u53cd': 251, '\u5fb7': 252, '\u6c34': 253, '\u5b9e': 254, '\u5747': 255, '\u7c73': 256, '\u53d8': 257, '\u6b21': 258, '\u63d0': 259, '\u5934': 260, '\u5c3c': 261, '\u963f': 262, 'E': 263, '\u8d27': 264, '\u697c': 265, '\u7a7a': 266, '\u52a1': 267, '\u8fc7': 268, '\u94f6': 269, '\u590d': 270, '\u683c': 271, '\u76f8': 272, '\u5b63': 273, '\u610f': 274, '\u597d': 275, '\u53cb': 276, '\u6027': 277, '\u5361': 278, '\u89e3': 279, '\u6298': 280, '\u514b': 281, '\u56e0': 282, '\u98ce': 283, '\u7ea2': 284, '\u540c': 285, '\u5e7f': 286, '\u5355': 287, '\u6237': 288, '\u5316': 289, '\u96be': 290, '\u5f55': 291, '\u66dd': 292, '\u533a': 293, '\u548c': 294, '\u62c9': 295, '\u8d70': 296, '\u51fb': 297, '\u5e08': 298, '\u4e50': 299, '\u6b3e': 300, '\u6f14': 301, '\u8bae': 302, '\u9898': 303, '\u544a': 304, '\u672a': 305, '\u677f': 306, '\u9662': 307, '\u521b': 308, 'M': 309, '\u53d6': 310, '\u578b': 311, '\u5236': 312, '\u4ee3': 313, '\u4f24': 314, '\u795e': 315, '\u5df2': 316, '\u52bf': 317, '\u6d3b': 318, '\u5230': 319, '\u5e94': 320, '\u6559': 321, '\u8d39': 322, '\u653e': 323, '\u58eb': 324, '\u81f3': 325, '\u770b': 326, '\u6211': 327, '\u7ed3': 328, '\u6740': 329, '\u8ba4': 330, '\u754c': 331, '\u5dde': 332, '\u67e5': 333, '\u76ee': 334, '\u8fbe': 335, '\u6797': 336, '\u513f': 337, '\u91cc': 338, '\u5411': 339, '\u9886': 340, '\u964d': 341, '\u5965': 342, '\u522b': 343, '\u7f57': 344, '\u7edf': 345, '\u9500': 346, '\u8fde': 347, '\u5c81': 348, '\u5f53': 349, '\u6b63': 350, '\u6848': 351, '\u8b66': 352, '\u534a': 353, '\u6d4b': 354, '\u53cc': 355, '\u5a5a': 356, '\u81f4': 357, '\u671b': 358, '\u95f4': 359, '\u5f0f': 360, '\u5f20': 361, '\u5c55': 362, '\u4eca': 363, '\u9053': 364, '\u82b1': 365, '\u4ea1': 366, '\u5f71': 367, '\u8fd1': 368, '\u7231': 369, '\u5ba2': 370, '\u91cf': 371, '\u4e89': 372, 'B': 373, '\u4f4d': 374, 'F': 375, '\u5931': 376, '\u4efb': 377, '\u767e': 378, '\u7248': 379, '\u7d22': 380, '\u6c42': 381, '\u5341': 382, '\u5c40': 383, '\u5bcc': 384, '\u7269': 385, '\u9645': 386, '\u7834': 387, '\u4e94': 388, '\u5f15': 389, '\u6b27': 390, '\u6216': 391, '\u8282': 392, '\u96c6': 393, '\u8f6c': 394, '\u6c5f': 395, '\u8272': 396, '\u9020': 397, '\u535a': 398, '\u7cfb': 399, '\u6240': 400, '\u9999': 401, '\u674e': 402, '\u7559': 403, '\u7ea6': 404, '\u6cb9': 405, '\u8fd0': 406, '\u5feb': 407, '\u5170': 408, '\u97e9': 409, '\u4ee5': 410, '\u56ed': 411, '\u72ec': 412, '\u5f97': 413, '\u9633': 414, '\u51a0': 415, '\u9f99': 416, '\u53ea': 417, 'l': 418, '\u6613': 419, '\u7206': 420, '\u5c11': 421, '\u6d41': 422, '\u804c': 423, '\u5c31': 424, '\u8d22': 425, '\u4e70': 426, '\u53f7': 427, '\u7591': 428, '\u56e2': 429, '\u8868': 430, '\u6539': 431, '\u8def': 432, '\u5bfc': 433, '\u8bed': 434, '\u5b9d': 435, '\u9001': 436, '\u5176': 437, '\u95ee': 438, '\u9707': 439, '\u4e8e': 440, '\u5e95': 441, '\u7247': 442, '\u62cd': 443, '\u611f': 444, '\u73a9': 445, '\u6807': 446, 'r': 447, '\u5956': 448, '\u592b': 449, '\u4ed6': 450, '\u89c4': 451, '\u4f55': 452, '\u4f18': 453, '\u4eb2': 454, '\u60f3': 455, '\u843d': 456, '\u9669': 457, '\u6caa': 458, '\u5212': 459, '\u51b3': 460, '\u53ca': 461, '\u5356': 462, '\u4f73': 463, '\u503c': 464, '\u5907': 465, '\u5b98': 466, '\u52a9': 467, '\u8db3': 468, '\u73af': 469, 'N': 470, '\u652f': 471, '\u53e3': 472, '\u5343': 473, '\u6392': 474, '\u5f39': 475, '\u6e05': 476, '\u51c6': 477, '\u8bb0': 478, 'V': 479, '\u6743': 480, '\u670d': 481, '\u8a00': 482, '\u7ba1': 483, '\u4ef6': 484, '\u5957': 485, '\u505a': 486, '\u771f': 487, '\u5fd7': 488, '\u6cb3': 489, '\u4eab': 490, '\u671d': 491, '\u9ec4': 492, '\u9648': 493, '\u4ec5': 494, '\u5f85': 495, '\u672f': 496, '\u63a5': 497, '\u539f': 498, '\u8c08': 499, '\u90fd': 500, '\u6574': 501, '\u62a2': 502, '\u6e56': 503, '\u798f': 504, '\u7ad9': 505, '\u80dc': 506, '\u5947': 507, '\u793a': 508, '\u79bb': 509, '\u529e': 510, '\u7b49': 511, '\u6b66': 512, '\u7acb': 513, '\u724c': 514, '\u6c14': 515, '\u8baf': 516, '\u6279': 517, '\u592a': 518, '\u663e': 519, '\u6ce8': 520, '\u4e3e': 521, '\u542f': 522, 'd': 523, '\u5a92': 524, '\u5982': 525, '\u66f4': 526, '\u9752': 527, '\u6280': 528, '\u5fae': 529, '\u79d8': 530, '\u5ba3': 531, '\u9ed1': 532, '\u503a': 533, '\u4ecd': 534, '\u5e9c': 535, '\u89c2': 536, '\u8d28': 537, '\u6a21': 538, '\u9632': 539, '\u4f0a': 540, '\u5885': 541, '\u4f4f': 542, '\u505c': 543, '\u9700': 544, '\u5e86': 545, '\u8fd8': 546, 'Q': 547, '\u5e45': 548, '\u5148': 549, '\u51cf': 550, '\u666e': 551, '\u975e': 552, '\u5426': 553, '\u5238': 554, '\u7387': 555, '\u7ee9': 556, 'X': 557, '\u60e0': 558, '\u878d': 559, '\u4e86': 560, '\u94b1': 561, '\u7ef4': 562, '\u51b2': 563, '\u4f01': 564, '\u9047': 565, '\u80b2': 566, '\u94c1': 567, '\u6551': 568, '\u5bc6': 569, '\u79c0': 570, '\u53c2': 571, '\u7b56': 572, '\u6d32': 573, '\u76f4': 574, '\u516d': 575, 's': 576, '\u793c': 577, '\u7533': 578, '\u679c': 579, '\u6df1': 580, '\u6625': 581, '\u6e90': 582, '\u6ce2': 583, '\u7968': 584, '\u89c1': 585, '\u6d88': 586, '\u536b': 587, '\u5b8c': 588, '\u4f60': 589, '\u7a81': 590, '\u66b4': 591, '\u606f': 592, '\u98de': 593, '\u5224': 594, '\u76db': 595, '\u7a0b': 596, '\u5e05': 597, '\u5047': 598, '\u666f': 599, '\u767b': 600, 'H': 601, '\u7701': 602, '\u6b4c': 603, '\u8bdd': 604, '\u8bbe': 605, '\u4fc4': 606, 't': 607, '\u6838': 608, '\u5373': 609, '\u5c1a': 610, '\u77ed': 611, '\u8bbf': 612, '\u529f': 613, '\u4f59': 614, '\u9891': 615, '\u8c6a': 616, '\u6e7e': 617, '\u4e91': 618, '\u77f3': 619, '\u767d': 620, '\u6ca1': 621, '\u73ed': 622, '\u653b': 623, '\u5b69': 624, '\u6d4e': 625, '\u7a76': 626, '\u8ba9': 627, '\u8f6e': 628, '\u7edd': 629, '\u53bb': 630, '\u4efd': 631, '\u538b': 632, '\u6bcd': 633, '\u4eae': 634, '\uff1f': 635, '\u7eb3': 636, '\u5e26': 637, '\u63a7': 638, '\u6cf0': 639, '\u7eaa': 640, '\u7b7e': 641, '\u521d': 642, '\u6b32': 643, '\u9650': 644, '\u7167': 645, '\u519c': 646, '\u8bba': 647, '\u901f': 648, '\u5de8': 649, '\u5267': 650, '\u5e55': 651, '\u6790': 652, '\u5174': 653, '\u76d1': 654, '\u75c5': 655, '\u8bfb': 656, '\u613f': 657, '\u77e5': 658, '\u4e66': 659, '\u5904': 660, '\u63ed': 661, '\u5218': 662, '\u82cf': 663, '\u5e2d': 664, '\u7565': 665, '\u7ee7': 666, '\u68a6': 667, 'R': 668, '\u5b58': 669, '\u9b54': 670, '\u62df': 671, '\u5e0c': 672, '\u7ec8': 673, 'W': 674, '\u534f': 675, '\u7ed9': 676, '\u59cb': 677, '\u50cf': 678, '\u8425': 679, '\u547d': 680, '\u8ff7': 681, '\u914d': 682, '\u5c01': 683, '\u66fe': 684, '\u4f17': 685, '\u6c47': 686, '\u4e5d': 687, '\u6b65': 688, '\u9000': 689, '\u6751': 690, '\u516b': 691, '\u725b': 692, '\u533b': 693, '\u593a': 694, '\u59bb': 695, '\u79fb': 696, '\u7edc': 697, '\u76df': 698, '\u5bb9': 699, '\u66fc': 700, '\u5a01': 701, '\u5370': 702, '\u4ece': 703, '\u9879': 704, '\u64ad': 705, '\u8bf7': 706, '\u6d3e': 707, '\u6e29': 708, '\u5211': 709, '\u52ab': 710, '\u89d2': 711, '\u5531': 712, 'h': 713, '\u9a8c': 714, 'K': 715, '\u72c2': 716, '\u786e': 717, '\u88ad': 718, '\u7ae0': 719, '\u8bf4': 720, 'm': 721, '\u7236': 722, '\u9152': 723, '\u5668': 724, '\u793e': 725, '\u8361': 726, '\u9a97': 727, '\u7c7b': 728, '\u827a': 729, '\u6599': 730, '\u4e49': 731, '\u4e34': 732, '\u676f': 733, '\u76ca': 734, '\u5c5e': 735, '\u5eb7': 736, '\u6ee1': 737, '\u5f81': 738, '\u5e97': 739, '\u822a': 740, '\u5199': 741, '\u4e60': 742, '\u4e5f': 743, '\u667a': 744, '\u6784': 745, '\u6fc0': 746, '\u8fdc': 747, '\u8ffd': 748, '\u72af': 749, '\u8d8a': 750, '\u79c1': 751, '\u4ed3': 752, '\u523a': 753, '\u9732': 754, '\u5371': 755, '\u5c9b': 756, '\u59d4': 757, '\u6bcf': 758, '\u4f7f': 759, '\u6d0b': 760, '\u5766': 761, '\u8bc9': 762, '\u79cd': 763, '\u6362': 764, '\u5e38': 765, '\u590f': 766, '\u5fc5': 767, '\u97f3': 768, '\u5217': 769, '\u5178': 770, '\u989d': 771, '\u52df': 772, '\u83f2': 773, '\u7ae5': 774, '\u6c49': 775, '\u6050': 776, 'c': 777, '\u8303': 778, '\u65e9': 779, '\u6cbb': 780, '\u7687': 781, '\u8f6f': 782, '\u8865': 783, '\u5609': 784, '\u571f': 785, '\u95fb': 786, '\u8350': 787, '\u627f': 788, '\u624d': 789, '\u62d2': 790, '\u8fce': 791, '\u7bee': 792, '\u70b8': 793, '\u9635': 794, '\u54cd': 795, '\u6781': 796, '\u5404': 797, '\u6bd2': 798, '\u5305': 799, '\u60ca': 800, '\u8857': 801, '\u6258': 802, '\u8f7b': 803, '\u53f2': 804, '\u6717': 805, '\u53e4': 806, '\u4e03': 807, '\u8428': 808, '\u601d': 809, '\u5f69': 810, '\u4e3d': 811, '\u526f': 812, '\u5f52': 813, '\u5251': 814, '\u7ec3': 815, '\u745e': 816, '\u5386': 817, '\u5012': 818, '\u9646': 819, '\u67aa': 820, '\u62ff': 821, '\u500d': 822, '\u5f88': 823, '\u592e': 824, '\u7a33': 825, '\u649e': 826, '\u8df3': 827, '\u6d6a': 828, '\u5b85': 829, '\u7855': 830, '\u8d25': 831, '\u56f4': 832, '\u9986': 833, '\u96c5': 834, '\u9003': 835, '\u7bad': 836, '\u5e01': 837, '\u8dd1': 838, '\u4fc3': 839, '\u6fb3': 840, '\u5348': 841, '\u58f0': 842, '\u94a2': 843, '\u643a': 844, '\u8c37': 845, '\u72b6': 846, '\u6b62': 847, '\u7ade': 848, '\u96c4': 849, '\u5e7b': 850, '\u950b': 851, '\u6001': 852, '\u68ee': 853, '\u4ee4': 854, '\u5e76': 855, '\u699c': 856, '\u8bfa': 857, '\u4f9b': 858, '\u90d1': 859, '\u7126': 860, '\u7d20': 861, '\u5171': 862, '\u591c': 863, '\u5ef6': 864, '\u88c1': 865, '\u6770': 866, '\u5b81': 867, '\u6052': 868, '\u5ba1': 869, 'y': 870, '\u65bd': 871, '\u7f6e': 872, '\u6d89': 873, '\u514d': 874, '\u5175': 875, '\u62a4': 876, '\u54e5': 877, '\u5ead': 878, '\u8d54': 879, '\u63a2': 880, '\u8fb9': 881, '\u6761': 882, '\u7136': 883, '\u7d27': 884, '\u8c01': 885, '\u8c61': 886, '\u753b': 887, '\u7eff': 888, '\u5cf0': 889, '\u8840': 890, '\u559c': 891, '\u5ff5': 892, '\u6469': 893, '\u96f7': 894, '\u8c22': 895, '\u4f26': 896, '\u76d7': 897, '\u636e': 898, '\u5f62': 899, '\u604b': 900, '\u9526': 901, '\u7f3a': 902, '\u7075': 903, '\u5f03': 904, '\u6f6e': 905, '\u5200': 906, '\u5883': 907, 'b': 908, '\u5b57': 909, '\u5987': 910, '\u7aef': 911, 'k': 912, '\u5077': 913, '\u4ed9': 914, '\u5bdf': 915, '\u6b22': 916, '\u7ffb': 917, '\u7532': 918, '\u987a': 919, '\u6da6': 920, '\u5c04': 921, '\u6837': 922, '\u51b0': 923, '\u836f': 924, '\u57df': 925, '\u6548': 926, '\u677e': 927, 'u': 928, '\u5acc': 929, '\u8bfe': 930, '\u5802': 931, '\u5207': 932, '\u4e92': 933, '\u8d74': 934, '\u7981': 935, '\u5757': 936, '\u517b': 937, '\u59da': 938, '\u8d23': 939, '\u84dd': 940, '\u5c42': 941, '\u9c81': 942, '\u665a': 943, '\u9f50': 944, '\u6676': 945, '\u51c0': 946, '\u65ad': 947, '\u65d7': 948, '\u76c8': 949, '\u4e30': 950, '\u8bad': 951, ' ': 952, '\u821e': 953, '\u5bd3': 954, '\u6768': 955, '\u9c9c': 956, '\u6311': 957, '\u5bfb': 958, '\u6839': 959, '\u4e25': 960, '\u6597': 961, '\u4eec': 962, '\u5409': 963, '\u7b11': 964, '\u6545': 965, '\u9ea6': 966, '\u5f8b': 967, '\u63f4': 968, '\u9876': 969, '\u7b80': 970, '\u5177': 971, '\u5b59': 972, '\u4fee': 973, '\u67b6': 974, '\u9738': 975, '\u900f': 976, '\u5dee': 977, '\uff01': 978, '\u8d1d': 979, '\u5e84': 980, '\u6025': 981, '\u773c': 982, '\u5b88': 983, '\u6653': 984, '\u68c0': 985, '\u6697': 986, '\u59d0': 987, '\u5854': 988, '\u660c': 989, '\u60a3': 990, '\u675f': 991, 'U': 992, '\u8bef': 993, '\u6c99': 994, '\u98df': 995, '\u4e39': 996, '\u54c8': 997, '\u5bf8': 998, '\u6267': 999, '\u8d37': 1000, '\u8be2': 1001, '\u6c38': 1002, '\u6000': 1003, '\u707e': 1004, '\u65c5': 1005, '\u6750': 1006, '\u7ec6': 1007, '\u6355': 1008, '\u4f1f': 1009, '\u903e': 1010, '\u627e': 1011, '\u8d62': 1012, 'p': 1013, '\u6863': 1014, '\u4fa0': 1015, '\u9519': 1016, '\u5377': 1017, '\u6297': 1018, '\u5434': 1019, '\u7a7f': 1020, '\u680b': 1021, '\u8d1f': 1022, '\u5df1': 1023, '\u5403': 1024, '\u955c': 1025, '\u7fa4': 1026, '\u6bb5': 1027, '\u5f31': 1028, '\u5ba4': 1029, '\u8fd9': 1030, '\u5065': 1031, '\u80cc': 1032, '\u96ea': 1033, '\u80e1': 1034, '\u79df': 1035, '\u5c3e': 1036, '\u85cf': 1037, '\u501f': 1038, '\u5f1f': 1039, '\u8fdd': 1040, '\u50a8': 1041, '\u7b97': 1042, '\u903c': 1043, '\u547c': 1044, '\u7b54': 1045, '\u4ec0': 1046, '\u5438': 1047, '\u5e72': 1048, '\u672b': 1049, '\u96e8': 1050, '\u4f19': 1051, '\u6885': 1052, '\u4f9d': 1053, '\u6682': 1054, '\u56f0': 1055, '\u7b14': 1056, '\u52d2': 1057, 'g': 1058, '\u65ec': 1059, '\u8ba8': 1060, '\u8c46': 1061, '\u90ed': 1062, '\u8bc8': 1063, '\u9a7e': 1064, '\u8fea': 1065, '\u517d': 1066, '\u949f': 1067, '\u7740': 1068, '\u5584': 1069, '\u94dc': 1070, '\u7b79': 1071, '\u8d35': 1072, '\u6269': 1073, '+': 1074, '\u515a': 1075, '\u9002': 1076, '\u9057': 1077, '\u5ddd': 1078, '\u7f5a': 1079, '\u53c8': 1080, '\u4e48': 1081, '\u7f29': 1082, '\u907f': 1083, '\u8d75': 1084, '\u4ed8': 1085, '\u987e': 1086, '\u79ef': 1087, '\u9759': 1088, '\u5360': 1089, '\u7259': 1090, '\u6f5c': 1091, '\u4e8f': 1092, '\u6293': 1093, '\u5723': 1094, '\u7389': 1095, '\u7cd6': 1096, '\u5e2e': 1097, '\u635f': 1098, '\u5b97': 1099, '\u8863': 1100, '\u83b1': 1101, '\u8d4f': 1102, '\u7801': 1103, '\u5760': 1104, '\u5c4a': 1105, '\u62b5': 1106, '\u62c5': 1107, '\u8fd4': 1108, '\u65e7': 1109, '\u7ec7': 1110, '\u5192': 1111, '\u6d1b': 1112, '\u79cb': 1113, '\u5bb3': 1114, '\u6c7d': 1115, '\u4ecb': 1116, '\u864e': 1117, '\u8bb2': 1118, 'J': 1119, '\u5c4f': 1120, '\u65cf': 1121, '\u7ebd': 1122, '\u66ff': 1123, '\u7092': 1124, '\u800c': 1125, '\u521a': 1126, '\u5145': 1127, '\u6ed1': 1128, '\u4f11': 1129, '\u9677': 1130, '\u5382': 1131, '\u6444': 1132, '\u8d5e': 1133, '\u66f2': 1134, '\u91c7': 1135, '\u7f6a': 1136, '\u805a': 1137, '\u952e': 1138, '\u6f2b': 1139, '\u4e01': 1140, '\u6069': 1141, '\u9547': 1142, '\u732e': 1143, '\u5bbe': 1144, '\u7a0e': 1145, '\u89e6': 1146, '\u73e0': 1147, '\u51b7': 1148, '\u9006': 1149, '\u8d5a': 1150, '\u6d25': 1151, '\u6db2': 1152, '\u8f93': 1153, '\u62e9': 1154, '\u5f02': 1155, '\u7adf': 1156, '\u8363': 1157, '\u7efc': 1158, '\u542c': 1159, '\u53ec': 1160, '\u4f8b': 1161, '\u5c3d': 1162, '\u7f18': 1163, '\u6c57': 1164, '\u8111': 1165, '\u8482': 1166, '\u5e93': 1167, '\u5708': 1168, '\u8d4c': 1169, '\u85aa': 1170, '\u5a31': 1171, '\u8239': 1172, '\u780d': 1173, '\u5ea7': 1174, '\u91ca': 1175, '\u7531': 1176, '\u7eb7': 1177, '\u718a': 1178, '\u82f9': 1179, '\u5e78': 1180, '\u9769': 1181, '\u62fc': 1182, '\u9644': 1183, '\u77ff': 1184, '\u6881': 1185, '\u9664': 1186, '\u5988': 1187, '\u575a': 1188, '\u8463': 1189, '\u59c6': 1190, '\u626c': 1191, '\u829d': 1192, '\u5e1d': 1193, '\u75af': 1194, '\u7b51': 1195, '\u654c': 1196, '\u6234': 1197, 'v': 1198, '\u575b': 1199, '\u624e': 1200, '\u5fe7': 1201, '\u4ea6': 1202, '\u8bb8': 1203, '\u6d2a': 1204, '\u8d8b': 1205, '\u811a': 1206, '\u52b2': 1207, '\u71d5': 1208, '\u9686': 1209, '\u5b8f': 1210, '\u68c9': 1211, '\u6012': 1212, '\u507f': 1213, '\u4e71': 1214, '\u64cd': 1215, '\u8be5': 1216, '\u4e08': 1217, '\u6d17': 1218, '\u987b': 1219, '\u5219': 1220, '\u8ff0': 1221, '\u8f7d': 1222, '\u7eaf': 1223, '\u5ba0': 1224, '\u5374': 1225, '\u5de7': 1226, '\u91ce': 1227, '\u6b23': 1228, '\u59b9': 1229, '\u62d8': 1230, '\u4f69': 1231, '\u628a': 1232, '\u963b': 1233, '\u9014': 1234, '\u83ab': 1235, '\u968f': 1236, '\u9a7b': 1237, '\u96f6': 1238, '\u6bb4': 1239, '\u6076': 1240, '\u7a83': 1241, '\u6bd5': 1242, '\u6563': 1243, '\u5b55': 1244, '\u9c7c': 1245, '\u62c6': 1246, '\u817e': 1247, '\u52c7': 1248, '\u60a6': 1249, '\u641c': 1250, '\u76ae': 1251, '\u6350': 1252, '\u9a91': 1253, '\u4fbf': 1254, '\u51ef': 1255, '\u9760': 1256, '\u6367': 1257, '\u7089': 1258, '\u8e2a': 1259, '\u4fb5': 1260, '\u5510': 1261, '\u8f9e': 1262, '\u7c89': 1263, '\u6982': 1264, '\u786c': 1265, '\u9080': 1266, '\u6dc0': 1267, '\u9177': 1268, '\u9690': 1269, '\u987f': 1270, '\u8bda': 1271, '\u6bc1': 1272, '\u9ece': 1273, '\u9010': 1274, '\u602a': 1275, '\u826f': 1276, '\u6389': 1277, '\u51e0': 1278, '\u9192': 1279, '\u6731': 1280, '\u80f6': 1281, '\u7cae': 1282, '\u5750': 1283, '\u6263': 1284, '\u504f': 1285, '\u76d6': 1286, '\u67cf': 1287, '\u586b': 1288, '\u5c38': 1289, '\u8be6': 1290, '\u95ed': 1291, '\u5fa1': 1292, '\u7f13': 1293, '\u6dd8': 1294, '\u585e': 1295, '\u6746': 1296, '\u4f2f': 1297, '\u71c3': 1298, 'Z': 1299, '\u4f3c': 1300, '\u7530': 1301, '\u95ea': 1302, '\u638c': 1303, '\u5f92': 1304, '\u6e10': 1305, '\u6d9b': 1306, '\u632b': 1307, '\u8bde': 1308, '\u5cb8': 1309, '\u4f30': 1310, '\u82af': 1311, '\u57c3': 1312, '\u62dc': 1313, '\u8c0b': 1314, '\u8de8': 1315, '\u8bc6': 1316, '\u6302': 1317, '\u5b9c': 1318, '\u51ac': 1319, '\u6167': 1320, '\u7a46': 1321, '\u7f72': 1322, '\u8000': 1323, '\u6d66': 1324, '\u8651': 1325, '\u6491': 1326, '\u969c': 1327, '\u7ed1': 1328, '\u9636': 1329, '\u642d': 1330, '\u5401': 1331, '\u4e1d': 1332, '\u82e5': 1333, '\u5978': 1334, '\u52b3': 1335, '\u5b8b': 1336, '\u82e6': 1337, '\u8d38': 1338, '\u4e4c': 1339, 'w': 1340, '\u70e7': 1341, '\u5473': 1342, '\u6d59': 1343, '\u89c8': 1344, '\u622a': 1345, '\u6696': 1346, '\u795d': 1347, '\u541b': 1348, '\u8131': 1349, '\u9605': 1350, '\u51b5': 1351, '\u9b42': 1352, '\u6388': 1353, '\u7164': 1354, '\u97e6': 1355, '\u8f83': 1356, '\u62d6': 1357, '\u98d9': 1358, '\u731b': 1359, '\u8273': 1360, '\u7d2b': 1361, '\u629b': 1362, '\u4e58': 1363, '\u5f90': 1364, '\u5de1': 1365, '\u90a3': 1366, '\u53eb': 1367, '\u7838': 1368, '\u88f8': 1369, '\u4e54': 1370, '\u6811': 1371, '\u5bab': 1372, '\u6668': 1373, '\u83dc': 1374, '\u8da3': 1375, '\u632f': 1376, '\u8ff9': 1377, '\u8fc5': 1378, '\u9093': 1379, '\u6ce1': 1380, '\u9910': 1381, '\u8230': 1382, '\u5733': 1383, '\u5706': 1384, '\u614e': 1385, '\u9189': 1386, '\u8584': 1387, '\u7433': 1388, '\u6cfd': 1389, '\u8058': 1390, '\u7eb2': 1391, '\u68d2': 1392, '\u8bcd': 1393, '\u90a6': 1394, '\u72f1': 1395, '\u6bdb': 1396, '\u6728': 1397, '\u827e': 1398, '\u5a1c': 1399, '\u7956': 1400, 'Y': 1401, '\u865a': 1402, '\u641e': 1403, '\u58a8': 1404, '\u7d2f': 1405, '\u6b64': 1406, '\u6c88': 1407, '\u732b': 1408, '\u8138': 1409, '\u9501': 1410, '\u6566': 1411, '\u75c7': 1412, '\u6cc4': 1413, '\u6de1': 1414, '\u7c4d': 1415, '\u9aa8': 1416, '\u7ea0': 1417, '\u5b87': 1418, '\u6021': 1419, '\u900a': 1420, '\u8d34': 1421, '\u8ba2': 1422, '\u6865': 1423, '\u4e61': 1424, '\u70c8': 1425, '\u5fd8': 1426, '\u8ddf': 1427, '\u57f9': 1428, '\u52b1': 1429, '\u6620': 1430, '\u5ef7': 1431, '\u70df': 1432, '\u4fca': 1433, '\u65a5': 1434, '\u5bbd': 1435, '\u70ae': 1436, '\u90ce': 1437, '\u8fdf': 1438, '\u5bb4': 1439, '\u73b2': 1440, '\u9706': 1441, '\u5019': 1442, '\u90ca': 1443, '\u6c89': 1444, '\u633a': 1445, '\u8dc3': 1446, '\u626e': 1447, '\u54ea': 1448, '\u7f16': 1449, '\u8f89': 1450, '\u675c': 1451, '\u8499': 1452, '\u523b': 1453, '\u6a2a': 1454, '\u8c28': 1455, '\u58c1': 1456, '\u8d4e': 1457, '\u7fd4': 1458, '\u8feb': 1459, '\u582a': 1460, '\u8349': 1461, '\u6dfb': 1462, '\u74e6': 1463, '\u996d': 1464, '\u5c24': 1465, '\u8bd7': 1466, '\u80ce': 1467, '\u5a74': 1468, '\u5e15': 1469, '\u6b8b': 1470, 'f': 1471, '\u6b49': 1472, '\u8d64': 1473, 'x': 1474, '\u6015': 1475, '\u9b3c': 1476, '\u574f': 1477, '\u62bc': 1478, '\u5f80': 1479, '\u8033': 1480, '\u7545': 1481, '\u9634': 1482, '\u8d60': 1483, '\u5b54': 1484, '\u62e5': 1485, '\u5c0a': 1486, '\u8f91': 1487, '\u5144': 1488, '\u5fc6': 1489, '\u6f58': 1490, '\u82d1': 1491, '\u6eaa': 1492, '\u56fa': 1493, '\u5bd2': 1494, '\u5c4b': 1495, '\u8f86': 1496, '\u62b1': 1497, '\u89c9': 1498, '\u626b': 1499, '\u6062': 1500, '\u75db': 1501, '\u6016': 1502, '\u5427': 1503, '\u5f17': 1504, '\u7597': 1505, '\u6572': 1506, '\u72d7': 1507, '\u60ac': 1508, '\u9a82': 1509, '\u53e6': 1510, '\u8d76': 1511, '\u7978': 1512, '\u8521': 1513, '\u7965': 1514, '\u8fc1': 1515, '\u4eea': 1516, '\u6d6e': 1517, '\u5e16': 1518, '\u8d29': 1519, '\u664b': 1520, '\u6cc9': 1521, '\u51f6': 1522, '\u4f34': 1523, '\u80af': 1524, '\u95f9': 1525, '\u654f': 1526, '\u68cb': 1527, '\u6df7': 1528, '\u9e4f': 1529, '\u67d3': 1530, '\u6380': 1531, '\u5a46': 1532, '\u6691': 1533, '\u8d3e': 1534, '\u6f0f': 1535, '\u8f68': 1536, '\u4e95': 1537, '\u6c83': 1538, '\u9e1f': 1539, '\u6241': 1540, '\u9ed8': 1541, '!': 1542, '\u8d26': 1543, '\u64a4': 1544, '\u8089': 1545, '\u814a': 1546, '\u6309': 1547, '\u6742': 1548, '\u5417': 1549, '\u5524': 1550, '\u4e45': 1551, '\u6b20': 1552, '\u53f6': 1553, '\u732a': 1554, '\u6446': 1555, '\u5976': 1556, '\u5e7c': 1557, '\u552f': 1558, '\u7eb8': 1559, '\u8e22': 1560, '\u7bc7': 1561, '\u970d': 1562, '\u4f0d': 1563, '\u8bf1': 1564, '\u8d3a': 1565, '\u8377': 1566, '\u7fc1': 1567, '\u8a89': 1568, '\u8fbd': 1569, '\u676d': 1570, '\u90e1': 1571, '\u966a': 1572, '\u751c': 1573, '\uff09': 1574, '\u518c': 1575, '\u65e6': 1576, '\u719f': 1577, '\u6148': 1578, '\u74dc': 1579, '\u62ab': 1580, '\u3001': 1581, '\u62d4': 1582, '\u6d1e': 1583, '\u5269': 1584, '\u8ddd': 1585, '\u670b': 1586, '\u60e8': 1587, '\u6d01': 1588, '\u820d': 1589, '\u730e': 1590, '\u559d': 1591, '\u5999': 1592, '\u5fd9': 1593, '\u4e9b': 1594, '\uff08': 1595, '\u70ab': 1596, '\u6447': 1597, '\u6316': 1598, '\u600e': 1599, '\u95ef': 1600, '\u60f9': 1601, '\u742a': 1602, '\u51af': 1603, '\u51e1': 1604, '\u7bb1': 1605, '\u5bbf': 1606, '\u4f5b': 1607, '\u5e9f': 1608, '\u6325': 1609, '\u6562': 1610, '\u8212': 1611, '\u7518': 1612, '\u5a18': 1613, '\u731c': 1614, '\u5be8': 1615, '\u53bf': 1616, '\u5362': 1617, '\u804a': 1618, '\u656c': 1619, '\u7ece': 1620, '\u54a8': 1621, '\u80c0': 1622, '\u90ae': 1623, '\u9038': 1624, '\u9881': 1625, '\u6c64': 1626, '\u5ec9': 1627, '\u9e3f': 1628, '\u80c1': 1629, '\u6cea': 1630, '\u5b64': 1631, '\u8f70': 1632, '\u5f79': 1633, '\u6270': 1634, '\u5a07': 1635, '\u5761': 1636, '\u4e88': 1637, '\u94dd': 1638, '\u63e1': 1639, '\u65a4': 1640, '\u77bb': 1641, '\u5de6': 1642, '\u5954': 1643, '\u4ec1': 1644, '\u70ad': 1645, '\u5ac1': 1646, '\u5179': 1647, '\u7275': 1648, '\u9b45': 1649, '\u888b': 1650, '\u7ecd': 1651, '\u8389': 1652, '\u6212': 1653, '\u8c31': 1654, '\u8a93': 1655, '\u690d': 1656, '\u5851': 1657, '\u9685': 1658, '\u5fbd': 1659, '\u9ebb': 1660, '\u7586': 1661, '\u5353': 1662, '\u5076': 1663, '\u9012': 1664, '\u6284': 1665, '\u60b2': 1666, '\u94bb': 1667, '\u4e27': 1668, '\u9f13': 1669, '\u80a5': 1670, '\u857e': 1671, '\u6e21': 1672, '\u51e4': 1673, '\u5b5f': 1674, '\u4fa3': 1675, '\u88c2': 1676, '\u7ffc': 1677, '\u8d3f': 1678, '\u6377': 1679, '\u59ae': 1680, '\u4faf': 1681, '\u67d4': 1682, '\u62d3': 1683, '\u8a79': 1684, '\u50ac': 1685, '\u917f': 1686, '\u73cd': 1687, '\u86cb': 1688, '\u6f84': 1689, '\u4f46': 1690, '\u62bd': 1691, '\u4e22': 1692, '\u817f': 1693, '\u72fc': 1694, '\u5272': 1695, '\u706d': 1696, '\u65ed': 1697, '\u79d2': 1698, '\u8c2d': 1699, '\u96c7': 1700, '\u5764': 1701, '\u63aa': 1702, '\u5428': 1703, '\u8457': 1704, '\u5bff': 1705, '\u6614': 1706, '\u6e20': 1707, '\u9f84': 1708, '\u9a9a': 1709, '\u75be': 1710, '\u904d': 1711, '\u8854': 1712, '\u6c6a': 1713, '\u8d2b': 1714, '\u8fa3': 1715, '\u6ee8': 1716, '\u5821': 1717, '\u96ef': 1718, '\u5e8f': 1719, '\u871c': 1720, '\u78a7': 1721, '\u9a70': 1722, '\u7eea': 1723, '\u86d9': 1724, '\u51cc': 1725, '\u8861': 1726, '\u6628': 1727, '\u67f1': 1728, '\u9274': 1729, '\u8c0d': 1730, '\u6458': 1731, '\u5448': 1732, '\u6602': 1733, '\u5151': 1734, '\u5385': 1735, '\u90bb': 1736, '\u946b': 1737, '\u608d': 1738, '\u8302': 1739, '\u9896': 1740, '\u520a': 1741, '\u80f8': 1742, '\u8f9b': 1743, '\u7761': 1744, '\u7238': 1745, '\u8bca': 1746, '\u9655': 1747, '\u5996': 1748, '\u9f0e': 1749, '\u6a61': 1750, '\u64bc': 1751, '\u8587': 1752, '\u74f6': 1753, '\u4e11': 1754, '\u60a8': 1755, '\u4f0f': 1756, '\u6deb': 1757, '\u8170': 1758, '\u5806': 1759, '\u76fc': 1760, '\u8d3c': 1761, '\u8896': 1762, '\u5835': 1763, '\u4f2a': 1764, '\u81e3': 1765, '\u82b3': 1766, '&': 1767, '\u8fc8': 1768, '\u9753': 1769, '\u4ed4': 1770, '\u60d1': 1771, '\u902e': 1772, '\u6c61': 1773, '\u8bdb': 1774, '\u9e70': 1775, '\u65fa': 1776, '\u4e52': 1777, '\u5fe0': 1778, '\u532a': 1779, '\u80de': 1780, '\u62d0': 1781, 'z': 1782, '\u542b': 1783, '\u5955': 1784, '\u6454': 1785, '\u6ea2': 1786, '\u5237': 1787, '\u5e8a': 1788, '\u543b': 1789, '\u6398': 1790, '\u695a': 1791, '\u6ce5': 1792, ']': 1793, '\u5948': 1794, '\u6b67': 1795, '\u55b7': 1796, '\u7070': 1797, '\u529d': 1798, '\u594b': 1799, '\u584c': 1800, '\u7eb9': 1801, '[': 1802, '\u98d8': 1803, '\u95f2': 1804, '\u4ff1': 1805, '\u517c': 1806, '\u75b2': 1807, '\u6500': 1808, '\u7eb5': 1809, '\u6c0f': 1810, '\u8352': 1811, '\u763e': 1812, '\u7fbd': 1813, '\u8d6b': 1814, '\u8fb0': 1815, '\u7f62': 1816, '\u6851': 1817, '\u6606': 1818, '\u7237': 1819, '\u79e6': 1820, '\u84c4': 1821, '\u61be': 1822, '\u4eff': 1823, '\u6345': 1824, '\u5c2c': 1825, '\u53a6': 1826, '\u8986': 1827, '\u60e7': 1828, '\u4ec7': 1829, '\u764c': 1830, '\u5c34': 1831, '\u54ed': 1832, '\u706f': 1833, '\u94fe': 1834, '\u503e': 1835, '\u6291': 1836, '\u591f': 1837, '\u9488': 1838, '\u96c0': 1839, '\u788d': 1840, '\u4ea8': 1841, '\u5439': 1842, '\u5899': 1843, '\u5dc5': 1844, '\u978b': 1845, '\u8461': 1846, '\u4e4f': 1847, '\u7a84': 1848, '\u62f3': 1849, '\u8102': 1850, '\u996e': 1851, '\u7199': 1852, '\u9e21': 1853, '\u6c9f': 1854, '\u5c16': 1855, '\u8f85': 1856, '\u9f20': 1857, '\u6478': 1858, '\u7235': 1859, '\u53f3': 1860, '\u524a': 1861, '\u7750': 1862, '\u9601': 1863, '\u5634': 1864, '\u50b2': 1865, '\u886b': 1866, '\u9676': 1867, '\u6652': 1868, '\u5979': 1869, '\u50f5': 1870, '\u8c0e': 1871, '\u82ac': 1872, '\u65c1': 1873, '\u7f8a': 1874, '\u9a76': 1875, '\u6c60': 1876, '\u5ae9': 1877, '\u558a': 1878, '\u548f': 1879, '\u6843': 1880, '\u9970': 1881, '\u915d': 1882, '\u5fcd': 1883, '\u7eef': 1884, '\u70bc': 1885, '\u59fb': 1886, '\u989c': 1887, '\u9065': 1888, '\u69db': 1889, '\u8f69': 1890, '\u6d8c': 1891, '\u8087': 1892, '\u8c23': 1893, '\u8e0f': 1894, '\u57cb': 1895, '\u6d69': 1896, '\u52ff': 1897, '\u8870': 1898, '\u632a': 1899, '\u8fb1': 1900, '\u6208': 1901, '\u950c': 1902, '\u63fd': 1903, '\u7a00': 1904, '\u9551': 1905, '\u9694': 1906, '\u6f9c': 1907, '\u68af': 1908, '\u5f0a': 1909, '\u7a97': 1910, '\u53e0': 1911, '\u7840': 1912, '\u6cf3': 1913, '\u66f9': 1914, '\u5c61': 1915, '\u4fa7': 1916, '\u6eda': 1917, '\u9a71': 1918, '\u94fa': 1919, '\u59dc': 1920, '\u5962': 1921, '\u723d': 1922, '\u8d2a': 1923, '\u6162': 1924, '\u53f9': 1925, '\u6cab': 1926, '\u7384': 1927, '\u6ee9': 1928, '\u574a': 1929, '\u8083': 1930, '\u7e41': 1931, '\u8c0a': 1932, '\u7a9d': 1933, '\u762b': 1934, '\u739b': 1935, '\u840c': 1936, '\u6108': 1937, '\u8bc0': 1938, '\u658c': 1939, '\u52e4': 1940, '\u5242': 1941, '\u916c': 1942, '\u8fa9': 1943, '\u73ae': 1944, '\u6e14': 1945, '\u821f': 1946, '\u59ff': 1947, '\u7763': 1948, '\u5f84': 1949, '\u642c': 1950, '\u7b26': 1951, '\u30fb': 1952, '\u5ffd': 1953, '\u88d4': 1954, '\u6296': 1955, '\u54ac': 1956, '\u6876': 1957, '\u5893': 1958, '\u5413': 1959, '\u88d5': 1960, '\u60dc': 1961, '\u788e': 1962, '\u8f7f': 1963, '\u6cca': 1964, '\u9a8f': 1965, '\u838e': 1966, '\u5783': 1967, '\u573e': 1968, '\u6124': 1969, '\u8036': 1970, '\u9e9f': 1971, '\u52aa': 1972, '\u62e8': 1973, '\u5220': 1974, '\u62e6': 1975, '\u6361': 1976, '\u539a': 1977, '\u51ed': 1978, '\u88d9': 1979, '\u6b8a': 1980, '\u5cb3': 1981, '\u90b5': 1982, '\u8d24': 1983, '\u7eb1': 1984, '\u7470': 1985, '\u9e7f': 1986, '\u5c97': 1987, '\u5915': 1988, '\u756a': 1989, '\u5195': 1990, '\u58f3': 1991, '\u64ce': 1992, '\u64d2': 1993, '\u9510': 1994, '\u8bf8': 1995, '\u6d53': 1996, '\u78b3': 1997, '\u70c2': 1998, '\u660f': 1999, '\u8150': 2000, '\u9022': 2001, '\u9a73': 2002, '\u6321': 2003, '\u6f02': 2004, '\u8eb2': 2005, '\u6bc5': 2006, '\u6b79': 2007, '\u6daf': 2008, '\u971e': 2009, '\u5a77': 2010, '\u6094': 2011, '\u4e73': 2012, '\u73ab': 2013, '\u6cbf': 2014, '\u5f7b': 2015, '\u6db5': 2016, '\u5bc4': 2017, '\u7f55': 2018, '\u8f74': 2019, '\u53d9': 2020, '\u6084': 2021, '\u810f': 2022, '\u2015': 2023, '\u5e7d': 2024, '\u5492': 2025, '\u8881': 2026, '\u7f34': 2027, '\u78b0': 2028, '\u53e5': 2029, '\u8c1c': 2030, '\u80a9': 2031, '\u75ab': 2032, '\u626d': 2033, '\u8179': 2034, '\u5de2': 2035, '\u731d': 2036, '\u8679': 2037, '\u5d14': 2038, '\u62fe': 2039, '\u7686': 2040, '\u502a': 2041, '\u85e4': 2042, '\u6028': 2043, '\u68b0': 2044, '\u5baa': 2045, '\u4e4e': 2046, '\u56da': 2047, '\u6656': 2048, '\u8dea': 2049, '\u8bd1': 2050, '\u65cb': 2051, '\u6bd9': 2052, '\u5a03': 2053, '\u5435': 2054, '\u9875': 2055, '\u5784': 2056, '\u8d56': 2057, '\u7784': 2058, '\u5146': 2059, '\u5e3d': 2060, '\u5c60': 2061, '\u594f': 2062, '\u5d1b': 2063, '\u722c': 2064, '\u8650': 2065, '\u65f1': 2066, '\u714c': 2067, '\u67ab': 2068, '\u574e': 2069, '\u72b9': 2070, '\u87ba': 2071, '\u94ed': 2072, '\u54f2': 2073, '\u6324': 2074, '\u901d': 2075, '\u5d29': 2076, '\uff0c': 2077, '\u51bb': 2078, '\u901b': 2079, '\u5c3a': 2080, '\u5154': 2081, '\u6c70': 2082, '\u6734': 2083, '\u5c1d': 2084, '\u4e32': 2085, '\u5a36': 2086, '\u90c1': 2087, '\u86c7': 2088, '\u789f': 2089, '\u6276': 2090, '\u540a': 2091, '\u9edb': 2092, '\u4fa6': 2093, '\u73ca': 2094, '\u8bbd': 2095, '\u6251': 2096, '\u8e29': 2097, '\u82d7': 2098, '\u6dc7': 2099, '\u7a3b': 2100, '\u5951': 2101, '\u614c': 2102, '\u6dbf': 2103, '\u7f44': 2104, '\u541e': 2105, '\u8336': 2106, '\u9631': 2107, '\u76fe': 2108, '\u848b': 2109, '\u8270': 2110, '\u6655': 2111, '\u76f2': 2112, '\u6eba': 2113, '\u88e4': 2114, '\u5eca': 2115, '\u9897': 2116, '\u67ef': 2117, '\u5ab3': 2118, '\u800d': 2119, '\u8702': 2120, '\u5b5d': 2121, '\u822c': 2122, '\u63a9': 2123, '\u8d4b': 2124, '\u80c6': 2125, '\u5141': 2126, '\u53db': 2127, '\u4fd7': 2128, '\u51f8': 2129, '\u5367': 2130, '\u9988': 2131, '\u5cad': 2132, '\u7ff0': 2133, '\u8f9f': 2134, '\u845b': 2135, '\u6254': 2136, '\u6ecb': 2137, '\u5395': 2138, '\u83b2': 2139, '\u52cb': 2140, '\u60ef': 2141, '\u644a': 2142, '\u8247': 2143, '\u68cd': 2144, '\u5986': 2145, '\u6cbd': 2146, '\u60d5': 2147, '\u6492': 2148, '\u6749': 2149, '\u9e45': 2150, '\u7c97': 2151, '\u5ca9': 2152, '\u9ca8': 2153, '\u7626': 2154, '\u6e58': 2155, '\u8d2f': 2156, '\u72ee': 2157, '\u78e8': 2158, '\u6d45': 2159, '\u6ede': 2160, '\u5415': 2161, '\u839e': 2162, '\u82cd': 2163, '\u70e6': 2164, '\u7cd5': 2165, '\u6842': 2166, '\u8328': 2167, '\u9aa4': 2168, '\u7529': 2169, '\u8427': 2170, '\u9493': 2171, '\u51f0': 2172, '\u62ac': 2173, '\u4fcf': 2174, '\u6b3a': 2175, '\u806a': 2176, '\u5c18': 2177, '\u8404': 2178, '\u6dd1': 2179, '\u4fa8': 2180, '\u5410': 2181, '\u8d21': 2182, '\u5203': 2183, '\u79a7': 2184, '\u61c2': 2185, '\u73bb': 2186, '\u8d81': 2187, '\u95f8': 2188, '\u5565': 2189, '\u51a4': 2190, '\u660a': 2191, '\u8c26': 2192, '\u77db': 2193, '\u846c': 2194, '\u6960': 2195, '\u8c34': 2196, '\u9178': 2197, '\u51c9': 2198, '\u777f': 2199, '\u78ca': 2200, '\u53a8': 2201, '\u5f66': 2202, '\u5632': 2203, '\u8010': 2204, '\u7a77': 2205, '\u8155': 2206, '\u9e23': 2207, '\u5339': 2208, '\u7ca4': 2209, '\u73d1': 2210, '\u7262': 2211, '\u62ef': 2212, '\u949e': 2213, '\u4f63': 2214, '\u7ed5': 2215, '\u640f': 2216, '\u8700': 2217, '\u5bb5': 2218, '\u4fe9': 2219, '\u6170': 2220, '\u60a0': 2221, '\u5de9': 2222, '\u8109': 2223, '\u7fe0': 2224, '\u77ac': 2225, '\u5085': 2226, '\u94c2': 2227, '\u63d2': 2228, '\u8017': 2229, '\u680f': 2230, '\u82ad': 2231, '\u75ea': 2232, '\u60bc': 2233, '\u6c1b': 2234, '\u684c': 2235, '\u6e34': 2236, '\u9c8d': 2237, '\u58f9': 2238, '\u67f3': 2239, '\u5021': 2240, '\u96d5': 2241, '\u7efd': 2242, '\u5740': 2243, '\u8367': 2244, '\u633d': 2245, '\u65a9': 2246, '\u575e': 2247, '\u809d': 2248, '\u69fd': 2249, '\u7325': 2250, '\u6e1d': 2251, '\u6cfc': 2252, '\u5f08': 2253, '\u6760': 2254, '\u526a': 2255, '\u5029': 2256, '\u6089': 2257, '\u5939': 2258, '\u7f69': 2259, '\u4e59': 2260, '\u5d07': 2261, '\u6beb': 2262, '\u7f9e': 2263, '\u7434': 2264, '\u8717': 2265, '\u5f95': 2266, '\u6349': 2267, '\u679a': 2268, '\u6d74': 2269, '\u6068': 2270, '\u76ef': 2271, '\u5256': 2272, '\u4ed5': 2273, '\u6e83': 2274, '\u78c5': 2275, '\u5949': 2276, '\u803b': 2277, '\u8096': 2278, '\u5578': 2279, '\u6664': 2280, '\u58ee': 2281, '\u743c': 2282, '\u7891': 2283, '\u796d': 2284, '\u708e': 2285, '\u80d6': 2286, '\u4eb5': 2287, '\u96fe': 2288, '\u840e': 2289, '\u57ab': 2290, ';': 2291, '\u54c0': 2292, '\u808b': 2293, '\u5c79': 2294, '\u99a8': 2295, '\u7af9': 2296, '\u6390': 2297, '\u9489': 2298, '\u5938': 2299, '\u5deb': 2300, '\u90b8': 2301, '\u73ba': 2302, '\u72ac': 2303, '\u7816': 2304, '\u70d9': 2305, '\u95f7': 2306, '\u7ed8': 2307, '\u6328': 2308, '\u98a0': 2309, '\u9524': 2310, '\u8be0': 2311, '\u851a': 2312, '\u6323': 2313, '\u5fcc': 2314, '\u4ef0': 2315, '\u4e5e': 2316, '\u8bbc': 2317, '\u67ec': 2318, '\u95f5': 2319, '\u6cdb': 2320, '\u5751': 2321, '\u866b': 2322, '\u7720': 2323, '\u9704': 2324, '\u5c91': 2325, '\u809a': 2326, '\u4f10': 2327, '\u5f70': 2328, '\u6735': 2329, '\u5c3f': 2330, '\u711a': 2331, '\u64e6': 2332, '\u4e7e': 2333, '\u8877': 2334, '\u72d0': 2335, '\u635e': 2336, '\u9556': 2337, '\u5f18': 2338, '\u4e43': 2339, '\u8f88': 2340, '\u888d': 2341, '\u4f83': 2342, '\u9521': 2343, '\u68da': 2344, '\u8776': 2345, '\u78be': 2346, '\u60e9': 2347, '\u59d3': 2348, '\u6f47': 2349, '\u97f5': 2350, '\u5bfa': 2351, '\u9b41': 2352, '\u5398': 2353, '\u8d30': 2354, '\u8f90': 2355, '\u9774': 2356, '\u7487': 2357, '\u7483': 2358, '\u859b': 2359, '\u54e8': 2360, '\u72e0': 2361, '\u918d': 2362, '\u9888': 2363, '\u6674': 2364, '\u9b4f': 2365, '\u5f6d': 2366, '\u9a84': 2367, '\u54c4': 2368, '\u6c76': 2369, '\u83c5': 2370, '\u9e92': 2371, '\u7f09': 2372, '\u80a4': 2373, '\u8e48': 2374, '\u906e': 2375, '\u7fce': 2376, '\u5cfb': 2377, '\u7845': 2378, '\u7985': 2379, '\u52a3': 2380, '\u94c5': 2381, '\u629a': 2382, '\u68d5': 2383, '\u59d1': 2384, '\u51f3': 2385, '\u63cf': 2386, '\u5937': 2387, '\u6ee5': 2388, '\u76d2': 2389, '\u66a8': 2390, '\u808c': 2391, '\u7f05': 2392, '\u8258': 2393, '\u5a9a': 2394, '\u5bc2': 2395, '\u5c6f': 2396, '\u4ed1': 2397, '\u7538': 2398, '\u679d': 2399, '\u7b3c': 2400, '\u5e06': 2401, '\u9505': 2402, '\u7f50': 2403, '\u7693': 2404, '\u7f1d': 2405, '\u7fa1': 2406, '\u80be': 2407, '\u7709': 2408, '\u8760': 2409, '\u90c5': 2410, '\u572d': 2411, '\u6101': 2412, '\u5ad6': 2413, '\u67dc': 2414, '\u84bf': 2415, '\u68e0': 2416, '\u8393': 2417, '\u537f': 2418, '\u8335': 2419, '\u7f14': 2420, '\u2161': 2421, '\u903b': 2422, '\u6bb7': 2423, '\u5c65': 2424, '\u5b99': 2425, '\u6342': 2426, '\u7c95': 2427, '\u7267': 2428, '\u8759': 2429, '\\u3000': 2430, '\u758f': 2431, '\u86ee': 2432, '\u75f4': 2433, '\u7a3f': 2434, '\u674f': 2435, '\u94a6': 2436, '\u7239': 2437, '\u997c': 2438, '\u4ef2': 2439, '\u6495': 2440, '\u90aa': 2441, '\u81c2': 2442, '\u84c9': 2443, '\u6b47': 2444, '\u5f25': 2445, '\u67d0': 2446, '\u7f20': 2447, '\u607c': 2448, '\u6e23': 2449, '\u4fde': 2450, '\u8f95': 2451, '\u8c79': 2452, '\u775b': 2453, '\u94a9': 2454, '\u52fe': 2455, '\u6988': 2456, '\u8d2c': 2457, '\u53d4': 2458, '\u7426': 2459, '\u5320': 2460, '\u7eba': 2461, '\u840d': 2462, '\u5f04': 2463, '\u5782': 2464, '\u70e8': 2465, '\u84b8': 2466, '\u63b7': 2467, '\u56e4': 2468, '\u7cdf': 2469, '\u5378': 2470, '\u7948': 2471, '\u80a2': 2472, '\u5e9a': 2473, '\u56ca': 2474, '\u4f88': 2475, '\u6714': 2476, '\u6d52': 2477, '\u9614': 2478, '\u84ec': 2479, '\u9042': 2480, '\u5983': 2481, '\u4f6c': 2482, '\u9f3b': 2483, '\u7504': 2484, '\u4ea9': 2485, '\u638f': 2486, '\u6ca6': 2487, '\u4f50': 2488, '\u6bef': 2489, '\u9091': 2490, '\u75d5': 2491, '\u8513': 2492, '\u6d19': 2493, '\u7792': 2494, '\u598d': 2495, '\u67f4': 2496, '\u6d3d': 2497, '\u4ed7': 2498, '\u7eee': 2499, '\u7a8d': 2500, '\u5589': 2501, '\u5f6c': 2502, '\u820c': 2503, '\u8c8c': 2504, '\u64c5': 2505, '\u7897': 2506, '\u6155': 2507, '\u7130': 2508, '\u8106': 2509, '\u6905': 2510, '\u4f1e': 2511, '\u70e4': 2512, '\u4fd0': 2513, '\u9cc4': 2514, '\u709c': 2515, '\u51fd': 2516, '\u6fa1': 2517, '\u57d4': 2518, '\u6850': 2519, '\u8305': 2520, '\u6977': 2521, '\u94a5': 2522, '#': 2523, '\u4fea': 2524, '\u621b': 2525, '\u8882': 2526, '\u83b9': 2527, '\u7280': 2528, '\u7624': 2529, '\u8fed': 2530, '\u4f38': 2531, '\u5ebe': 2532, '\u841d': 2533, '\u59a5': 2534, '\u5564': 2535, '\u5566': 2536, '\u54b8': 2537, '\u538c': 2538, '\u5e9e': 2539, '\u611a': 2540, '\u5bb0': 2541, '\u53c9': 2542, '\u621f': 2543, '\u94a7': 2544, '\u6168': 2545, '\u8292': 2546, '\u75bc': 2547, '\u4f51': 2548, '\u89c5': 2549, '\u7f24': 2550, '\u80bf': 2551, '\u6591': 2552, '\u64ac': 2553, '\u6e0a': 2554, '\u5eb8': 2555, '\u5239': 2556, '\u50e7': 2557, '\u8d1e': 2558, '\u6127': 2559, '\u6f20': 2560, '\u522e': 2561, '\u4e9f': 2562, '\u904f': 2563, '\u5c48': 2564, '\u6d12': 2565, '\u867d': 2566, '\u609f': 2567, '\u631f': 2568, '\u9e2d': 2569, '\u6247': 2570, '\u8fa8': 2571, '\u745c': 2572, '\u7fc5': 2573, '\u5c39': 2574, '\u655b': 2575, '\u9063': 2576, '\u533f': 2577, '\u5a55': 2578, '\u80ba': 2579, '\u5cd9': 2580, '\u6bbf': 2581, '\u5ce1': 2582, '\u5a49': 2583, '\u7ee3': 2584, '\u7272': 2585, '\u8c6b': 2586, '\u7b52': 2587, '\u64b0': 2588, '\u4f84': 2589, '\u7bea': 2590, '\u5974': 2591, '\u5669': 2592, '\u5446': 2593, '\u7476': 2594, '\u7334': 2595, '\u6d85': 2596, '\u5a7f': 2597, '\u72d9': 2598, '\u70eb': 2599, '\u90dd': 2600, '\u8df5': 2601, '\u987d': 2602, '\u57ae': 2603, '\u5e42': 2604, '\u831c': 2605, '\u819c': 2606, '\u8dfb': 2607, '\u6846': 2608, 'j': 2609, '\u7554': 2610, '\u6346': 2611, '\u6e24': 2612, '\u5dfe': 2613, '\u6d82': 2614, '\u6b96': 2615, '\u754f': 2616, '\u8e7f': 2617, '\u9e64': 2618, '\u527f': 2619, '\u5a9b': 2620, '\u6c40': 2621, '\u535c': 2622, '\u94f8': 2623, '\u6401': 2624, '\u78cb': 2625, '\u7ef3': 2626, '\u4fef': 2627, '\u8042': 2628, '\u8c10': 2629, '\u9ccc': 2630, '\u9675': 2631, '\u7d6e': 2632, '\u59a8': 2633, '\u90b1': 2634, '\u594e': 2635, '\u6670': 2636, '\u727a': 2637, '\u725f': 2638, '\u7a74': 2639, '\u8403': 2640, '\u501a': 2641, '\u8be1': 2642, '\u6c27': 2643, '\u8d31': 2644, '\u6064': 2645, '\u819d': 2646, '\u4e14': 2647, '\u704c': 2648, '\u54d1': 2649, '\u5ed6': 2650, '\u5f7c': 2651, '\u6df3': 2652, '\u6405': 2653, '\u5e10': 2654, '\u5308': 2655, '\u971c': 2656, '\u7b08': 2657, '\u7ea4': 2658, '\u80a0': 2659, '\u914b': 2660, '\u884d': 2661, '\u5d16': 2662, '\u74f7': 2663, '\u621a': 2664, '\u8845': 2665, '\u8180': 2666, '\u7c7c': 2667, '\u5ac2': 2668, 'q': 2669, '\u83cc': 2670, '\u701a': 2671, '\u6ce3': 2672, '\u9187': 2673, '\u953b': 2674, '\u71ac': 2675, '\u5384': 2676, '\u8749': 2677, '\u7cb9': 2678, '\u7490': 2679, '\u625b': 2680, '\u88f9': 2681, '\u832b': 2682, '\u78c1': 2683, '\u5389': 2684, '\u4f5f': 2685, '\u62ec': 2686, '\u94b0': 2687, '\u8f67': 2688, '\u4f58': 2689, '\u662d': 2690, '\u8339': 2691, '\u5b83': 2692, '\u952f': 2693, '\u67c4': 2694, '\u9e20': 2695, '\u8046': 2696, '\u9971': 2697, '\u76d0': 2698, '\u745f': 2699, '\u4ead': 2700, '\u6d4f': 2701, '\u6ef4': 2702, '\u8d48': 2703, '\u5618': 2704, '\u5319': 2705, '\u5f98': 2706, '\u5f8a': 2707, '\u7aa5': 2708, '\u6e7f': 2709, '\u5960': 2710, '\u9716': 2711, '\u5bde': 2712, '\u962e': 2713, '\u7b1b': 2714, '\u62c7': 2715, '\u9882': 2716, '\u6273': 2717, '\u61d2': 2718, '\u50bb': 2719, '\u5faa': 2720, '\u95eb': 2721, '\u786b': 2722, '\u80c3': 2723, '\u5c7f': 2724, '\u5582': 2725, '\u94c9': 2726, '\u9550': 2727, '\u90dc': 2728, '\u03b1': 2729, '\u4e1e': 2730, '\u74a8': 2731, '\u7410': 2732, '\u8715': 2733, '\u4f3a': 2734, '\u5496': 2735, '\u59a9': 2736, '\u574d': 2737, '\u8c05': 2738, '\u8f76': 2739, '\u8d43': 2740, '\u6cb8': 2741, '\u9893': 2742, '\u51b6': 2743, '\u5580': 2744, '\u6252': 2745, '\u59ec': 2746, '\u5858': 2747, '\u5f64': 2748, '\u97ad': 2749, '\u6556': 2750, '\u5ab2': 2751, '\u6df9': 2752, '\u79fd': 2753, '\u6b6a': 2754, '\u997f': 2755, '\u6a59': 2756, '\u9a74': 2757, '\u6c9b': 2758, '\u70fd': 2759, '\u8d41': 2760, '\u7480': 2761, '\u6fd2': 2762, '\u5993': 2763, '\u60df': 2764, '\u86db': 2765, '\u749e': 2766, '\u51d1': 2767, '\u81fb': 2768, '\u548b': 2769, '\u964c': 2770, '\u5561': 2771, '\u6816': 2772, '\u7b4b': 2773, '\u9ad3': 2774, '\u8c5a': 2775, '\u77f6': 2776, '\u817b': 2777, '\u96cd': 2778, '\u8116': 2779, '\u7ef5': 2780, '\u7a79': 2781, '\u9761': 2782, '\u6893': 2783, '\u7329': 2784, '\u65e2': 2785, '\u90b9': 2786, '\u6467': 2787, '\u9075': 2788, '\u58f6': 2789, '\u70ac': 2790, '\u66e6': 2791, '\u9887': 2792, '\u8343': 2793, '\u515c': 2794, '\u5d0e': 2795, '\u8154': 2796, '\u5288': 2797, '\u52c3': 2798, '\u5f2f': 2799, '\u72c4': 2800, '\u6627': 2801, '\u634d': 2802, '\u9f9a': 2803, '\u576f': 2804, '\u715e': 2805, '\u6289': 2806, '\u7115': 2807, '\u659c': 2808, '\u9f9f': 2809, '\u8ba7': 2810, '\u606d': 2811, '\u81ed': 2812, '\u6984': 2813, '\u5b9b': 2814, '\u707f': 2815, '\u8015': 2816, '\u5a25': 2817, '\u62a8': 2818, '\u6615': 2819, '\u5507': 2820, '\u5e99': 2821, '\u6ed5': 2822, '\u79bd': 2823, '\u97f6': 2824, '\u4e19': 2825, '\u83f1': 2826, '\u9017': 2827, '\u7eda': 2828, '\u8bc5': 2829, '\u821c': 2830, '\u66d9': 2831, '\u6a44': 2832, '\u8718': 2833, '\u8086': 2834, '\u94f2': 2835, '\u82b8': 2836, '\u9cb8': 2837, '\u9b44': 2838, '\u63b4': 2839, '\u9e93': 2840, '\u70b3': 2841, '\u7eb6': 2842, '\u751a': 2843, '\u9890': 2844, '\u65a7': 2845, '\u900d': 2846, '\u69a8': 2847, '\u5815': 2848, '\u7cca': 2849, '\u4e56': 2850, '\u7779': 2851, '\u76c6': 2852, '\u599e': 2853, '\u6a80': 2854, '\u5c90': 2855, '\u81ba': 2856, '\u8431': 2857, '\u8c13': 2858, '\u7076': 2859, '\u9102': 2860, '\u576a': 2861, '\u6b89': 2862, '\u554a': 2863, '\u9523': 2864, '\u9e26': 2865, '\u8e81': 2866, '\u64c2': 2867, '\u8f99': 2868, '\u51dd': 2869, '\u66a7': 2870, '\u81c0': 2871, '\u785d': 2872, '\u8c24': 2873, '\u5188': 2874, '\u5be1': 2875, '\u8165': 2876, '\u5f9c': 2877, '\u5f89': 2878, '\u8774': 2879, '\u53a2': 2880, '\u62e2': 2881, '\u77ee': 2882, '\u94c0': 2883, '\u8beb': 2884, '\u7afa': 2885, '\u8822': 2886, '\u7a57': 2887, '\u5c9a': 2888, '\u5a3c': 2889, '\u7f81': 2890, '\u6da9': 2891, '\u8bfd': 2892, '\u9e3d': 2893, '\u5265': 2894, '\u55dc': 2895, '\u889c': 2896, '\u626f': 2897, '\u836b': 2898, '\u78d5': 2899, '\u8309': 2900, '\u739f': 2901, '\u79be': 2902, '\u54aa': 2903, '\u5c41': 2904, '\u5824': 2905, '\u7a0d': 2906, '\u94c3': 2907, '\u5f11': 2908, '\uff05': 2909, '\u69b4': 2910, '\u592d': 2911, '\u960e': 2912, '\u575d': 2913, '\u5112': 2914, '\u59e5': 2915, '\u6f06': 2916, '\u7cbd': 2917, '\u978d': 2918, '\u79b9': 2919, '\u9965': 2920, '\u54ee': 2921, '\u804b': 2922, '\u98da': 2923, '\u8206': 2924, '\u949b': 2925, '=': 2926, '\u9502': 2927, '\u7aa6': 2928, '\u62b9': 2929, '\u6fe0': 2930, '\u8680': 2931, '\u7aed': 2932, '\u4e1b': 2933, '\u543c': 2934, '\u6073': 2935, '\u5366': 2936, '\u6f6d': 2937, '\u81e7': 2938, '\u9171': 2939, '\u9541': 2940, '\u683d': 2941, '\u8d42': 2942, '\u62f7': 2943, '\u6cfb': 2944, '\u6ba1': 2945, '\u5f26': 2946, '\u68a8': 2947, '\u58e4': 2948, '\u6c79': 2949, '\u65e8': 2950, '\u68fa': 2951, '\u7a3d': 2952, '\u5180': 2953, '\u6d47': 2954, '\u7352': 2955, '\u7c92': 2956, '\u6d5a': 2957, '\u93d6': 2958, '\u948a': 2959, '\u716e': 2960, '\u8574': 2961, '\u6512': 2962, '\u527d': 2963, '\u7f06': 2964, '\u5189': 2965, '\u77a9': 2966, '\u5e37': 2967, '\u778e': 2968, '\u9082': 2969, '\u9005': 2970, '\u9756': 2971, '\u7a9f': 2972, '\u94d0': 2973, '\u77e9': 2974, '\u8eba': 2975, '\u8299': 2976, '\u813e': 2977, '\u8e4a': 2978, '\u63a0': 2979, '\u8098': 2980, '\u61a7': 2981, '\u61ac': 2982, '\u6e9c': 2983, '\u8235': 2984, '\u5a1f': 2985, '\u7fd8': 2986, '\u79e9': 2987, '\u8205': 2988, '\u8721': 2989, '\u83c1': 2990, '\u5631': 2991, '\u5ce5': 2992, '\u4e18': 2993, '\u66ae': 2994, '\u918b': 2995, '\u6d46': 2996, '\u77e2': 2997, '\u997d': 2998, '\u5792': 2999, '\u90e6': 3000, '\u5486': 3001, '\u5f6a': 3002, '\u901e': 3003, '\u5be5': 3004, '\u973e': 3005, '\u7fe1': 3006, '\u8f9c': 3007, '\u7eac': 3008, '\u968b': 3009, '\u849c': 3010, '\u6643': 3011, '\u9985': 3012, '\u964b': 3013, '\u9f7f': 3014, '\u9610': 3015, '\u9ab8': 3016, '\u714e': 3017, '\u98d3': 3018, '\u8df7': 3019, '\u8d61': 3020, '\u4e38': 3021, '\u7fcc': 3022, '\u6320': 3023, '\u853d': 3024, '\u8e1d': 3025, '\u8f8d': 3026, '\u5351': 3027, '\u73c0': 3028, '\u4fed': 3029, \"'\": 3030, '\u6c41': 3031, '\u8bf5': 3032, '\u8306': 3033, '\u81a8': 3034, '\u7a92': 3035, '\u03bc': 3036, '\u7a98': 3037, '\u6233': 3038, '\u61c8': 3039, '\u75a4': 3040, '\u704f': 3041, '\u79c9': 3042, '\u54d7': 3043, '\u5e87': 3044, '\u6986': 3045, '\u6d3c': 3046, '\u7802': 3047, '\u6853': 3048, '\u8231': 3049, '\u5475': 3050, '\u92ae': 3051, '\u5c82': 3052, '\u4e4d': 3053, '\u87f9': 3054, '\u6dcb': 3055, '\u7a9c': 3056, '\u665f': 3057, '\u6995': 3058, '\u5777': 3059, '\u6a31': 3060, '\u6d78': 3061, '\u7caa': 3062, '\u6912': 3063, '\u95fa': 3064, '\u7901': 3065, '\u73cf': 3066, '\u97e7': 3067, '\u7396': 3068, '\u7c7d': 3069, '\u8bec': 3070, '\u655e': 3071, '\u7eca': 3072, '\u4e53': 3073, '\u817a': 3074, '\u91dc': 3075, '\u7be1': 3076, '\u6631': 3077, '\u6dee': 3078, '\u627c': 3079, '\u7fe9': 3080, '\u8559': 3081, '\u6590': 3082, '\u892a': 3083, '\u52c9': 3084, '\u67ad': 3085, '\u6789': 3086, '\u575f': 3087, '\u6487': 3088, '\u8537': 3089, '\u8346': 3090, '\u5f13': 3091, '\u5dcd': 3092, '\u6b83': 3093, '\u7b28': 3094, '\u6e3a': 3095, '\u950f': 3096, '\u8681': 3097, '\u909d': 3098, '\u5992': 3099, '\u6a58': 3100, '\u9eef': 3101, '\u4f3d': 3102, '\u6b46': 3103, '\u5243': 3104, '\u9972': 3105, '\u5df7': 3106, '\u6c5d': 3107, '\u7f2a': 3108, '\u59e8': 3109, '\u6cbe': 3110, '\u54e9': 3111, '\u8e6d': 3112, '\u867e': 3113, '\u87d2': 3114, '\u717d': 3115, '\u68ad': 3116, '\u84b2': 3117, '\u9709': 3118, '\u68f5': 3119, '\u53a5': 3120, '\u9a87': 3121, '\u7ad6': 3122, '\u622e': 3123, '\u5d69': 3124, '\u7b8d': 3125, '\u682a': 3126, '\u7194': 3127, '\u5ac9': 3128, '\u55d3': 3129, '\u747e': 3130, '\u664f': 3131, '\u8fe5': 3132, '\u601c': 3133, '\u5a06': 3134, '\u7184': 3135, '\u9975': 3136, '\u5026': 3137, '\u589f': 3138, '\u6fee': 3139, '\u6e17': 3140, '\u62fd': 3141, '\u609a': 3142, '\u962a': 3143, '\u94ce': 3144, '\u9a6d': 3145, '\u5cea': 3146, '\u5323': 3147, '\u854a': 3148, '\u7f22': 3149, '\u9e33': 3150, '\u9e2f': 3151, '\u9497': 3152, '\u803d': 3153, '\u55bb': 3154, '\u6402': 3155, '\u6d51': 3156, '\u5671': 3157, '\u8c41': 3158, '\u90a2': 3159, '\u9976': 3160, '\u5e62': 3161, '\u4f7c': 3162, '\u61ff': 3163, '\u8d66': 3164, '\u5a05': 3165, '\u5b5a': 3166, '\u88f4': 3167, '\u4fd8': 3168, '\u7b50': 3169, '\u54bd': 3170, '\u4f36': 3171, ',': 3172, '\u6eaf': 3173, '\u5ba5': 3174, '\u7b5d': 3175, '\u4ea5': 3176, '\u5bdd': 3177, '\u6a0a': 3178, '\u7085': 3179, '\u9e6d': 3180, '\u9163': 3181, '\u715c': 3182, '\u8169': 3183, '\u5e1c': 3184, '\u886c': 3185, '\u7977': 3186, '\u5506': 3187, '\u82db': 3188, '\u634f': 3189, '\u595a': 3190, '\u5ae3': 3191, '\u6f4d': 3192, '\u8c1b': 3193, '\u6c30': 3194, '\u4e10': 3195, '\u6795': 3196, '\u7409': 3197, '\u6363': 3198, '\u5201': 3199, '\u9668': 3200, '\u69cc': 3201, '\u5501': 3202, '\u8e66': 3203, '\u7f07': 3204, '\u5703': 3205, '\u5bb8': 3206, '\u55a7': 3207, '\u96cf': 3208, '\u75d2': 3209, '\u5306': 3210, '\u543e': 3211, '\u852c': 3212, '\u82a6': 3213, '\u9576': 3214, '\u5d2d': 3215, '\u7656': 3216, '\u766b': 3217, '\u78b1': 3218, '\u8e44': 3219, '\u6ca7': 3220, '\u53ee': 3221, '\u6194': 3222, '\u60b4': 3223, '\u810a': 3224, '\u9661': 3225, '\u5a29': 3226, '\u67af': 3227, '\u51a5': 3228, '\u970f': 3229, '\u9619': 3230, '\u6f5e': 3231, '\u52d8': 3232, '\u63cd': 3233, '\u5b09': 3234, '\u80fa': 3235, '\u96a7': 3236, '\u6070': 3237, '\u631d': 3238, '\u6dfc': 3239, '\u566a': 3240, '\u63ea': 3241, '\u7aff': 3242, '\u7ed2': 3243, '\u7252': 3244, '\u6942': 3245, '\u8e39': 3246, '\u7578': 3247, '\u9877': 3248, '\u7ef0': 3249, '\u73de': 3250, '\u8f97': 3251, '\u83ca': 3252, '\u835f': 3253, '\u5bc7': 3254, '\u9885': 3255, '\u7292': 3256, '\u77b0': 3257, '\u8bb6': 3258, '\u6687': 3259, '\u6400': 3260, '\u6c8c': 3261, '\u8f96': 3262, '\u50da': 3263, '\u7566': 3264, '\u7ec5': 3265, '\u8e0a': 3266, '\u4fae': 3267, '\u7c98': 3268, '\u8e35': 3269, '\u54c9': 3270, '\u7fdf': 3271, '\u8e72': 3272, '\u70bd': 3273, '\u6cae': 3274, '\u72e9': 3275, '\u8db4': 3276, '\u6979': 3277, '\u9a81': 3278, '\u7ede': 3279, '\u7525': 3280, '\u8de4': 3281, '\u5543': 3282, '\u752b': 3283, '\u6096': 3284, '\u566c': 3285, '\u62a0': 3286, '\u60fa': 3287, '\u88b1': 3288, '\u84d3': 3289, '\u868a': 3290, '\u6da7': 3291, '\u7ca5': 3292, '\u6151': 3293, '\u5c8c': 3294, '\u573b': 3295, '\u5254': 3296, '\u6619': 3297, '\u69df': 3298, '\u89ca': 3299, '\u89ce': 3300, '\u5b5c': 3301, '\u60eb': 3302, '\u7a91': 3303, '\u618b': 3304, '\u7c3f': 3305, '\u8782': 3306, '\u74a7': 3307, '\u6756': 3308, '\u6b87': 3309, '\u5a34': 3310, '\u63e3': 3311, '\u5375': 3312, '\u7f79': 3313, '\u5598': 3314, '\u5b40': 3315, '\u8ba5': 3316, '\u6cd3': 3317, '\u9508': 3318, '\u7737': 3319, '\u70f9': 3320, '\u7ef8': 3321, '\u82ae': 3322, '\u6cef': 3323, '\u7fb2': 3324, '\u8d63': 3325, '\u5c4e': 3326, '\u7131': 3327, '\u6e85': 3328, '\u5587': 3329, '\u67a2': 3330, '\u5984': 3331, '\u865e': 3332, '\u6d63': 3333, '\u7bf7': 3334, '\u56a3': 3335, '\u9699': 3336, '\u5768': 3337, '\u66dc': 3338, '\u7bd1': 3339, '\u548e': 3340, '\u673d': 3341, '\u77eb': 3342, '\u5ae6': 3343, '\u98a4': 3344, '\u5e3c': 3345, '\u7425': 3346, '\u8eac': 3347, '\u722a': 3348, '\u9a6f': 3349, '\u6da1': 3350, '\u5b95': 3351, '\u707c': 3352, '\u818f': 3353, '\u2103': 3354, '\u5fcf': 3355, '\u6a1f': 3356, '\u7ef7': 3357, '\u4f8d': 3358, '\u4ec6': 3359, '\u9122': 3360, '\u9a86': 3361, '\u516e': 3362, '\u8d9f': 3363, '\u8331': 3364, '\u828b': 3365, '\u6c13': 3366, '\u741b': 3367, '\u68b3': 3368, '\u4e2b': 3369, '\u803f': 3370, '\u76f9': 3371, '\u70db': 3372, '\u9573': 3373, '\u53e9': 3374, '\u788c': 3375, '\u7cd9': 3376, '\u6866': 3377, '\u8d50': 3378, '\u86f0': 3379, '\u6dae': 3380, '\u4f8f': 3381, '\u5d4c': 3382, '\u6c90': 3383, '\u94be': 3384, '\u54b1': 3385, '\u5b6a': 3386, '\u9600': 3387, '\u72f8': 3388, '\u9739': 3389, '\u7011': 3390, '\u62cc': 3391, '\u9b47': 3392, '\u7172': 3393, '\u6c55': 3394, '\u59aa': 3395, '\u59d7': 3396, '\u818a': 3397, '\u7ae3': 3398, '\u6c81': 3399, '\u54fd': 3400, '\u4ee8': 3401, '\u6ed4': 3402, '\u7984': 3403, '\u8327': 3404, '\u7cd7': 3405, '\u6c2f': 3406, '\u8475': 3407, '\u62ce': 3408, '\u5315': 3409, '\u78d0': 3410, '\u03c0': 3411, '\u52fa': 3412, '\u2236': 3413, '\u62d9': 3414, '\u7f38': 3415, '\u592f': 3416, '\u5f29': 3417, '\u76d4': 3418, '\u9a7f': 3419, '\u82b9': 3420, '\u53ed': 3421, '\u96bc': 3422, '\u6a3e': 3423, '\u9e25': 3424, '\u8236': 3425, '\u85af': 3426, '\u6c5b': 3427, '\u8446': 3428, '\u6b92': 3429, '\u8bb3': 3430, '\u6817': 3431, '\u7682': 3432, '\u755c': 3433, '\u745b': 3434, '\u74d2': 3435, '\u63b3': 3436, '\u90ac': 3437, '\u7a96': 3438, '\u8fc4': 3439, '\u5f99': 3440, '\u8d5d': 3441, '\u7a88': 3442, '\u7a95': 3443, '\u4f70': 3444, '\u96f3': 3445, '\u5ae1': 3446, '\u63b0': 3447, '\u6e44': 3448, '\u97a0': 3449, '\u8695': 3450, '\u6635': 3451, '\u9157': 3452, '\u70c1': 3453, '\u77e3': 3454, '\u8c0f': 3455, '\u70ef': 3456, '\u9a8b': 3457, '\u5cac': 3458, '\u9798': 3459, '\u9175': 3460, '\u65f7': 3461, '\u55e8': 3462, '\u631a': 3463, '\u8471': 3464, '\u5520': 3465, '\u6cbc': 3466, '\u5d58': 3467, '\u80e4': 3468, '\u8722': 3469, '\u8426': 3470, '\u607a': 3471, '\u6cf5': 3472, '\u949d': 3473, '\u658b': 3474, '\u86ca': 3475, '\u7455': 3476, '\u545c': 3477, '\u7f9a': 3478, '\u83c7': 3479, '\u5f57': 3480, '\u5b75': 3481, '\u5c51': 3482, '\u8304': 3483, '\u6577': 3484, '\u6059': 3485, '\u5a04': 3486, '\u8549': 3487, '\u62f1': 3488, '\u8783': 3489, '\u6897': 3490, '\u9e4a': 3491, '\u75ca': 3492, '\u8c2c': 3493, '\u6177': 3494, '\u51ff': 3495, '\u564e': 3496, '\u5fd0': 3497, '\u5fd1': 3498, '\u5471': 3499, '\u72ed': 3500, '\u77bf': 3501, '\u7ba9': 3502, '\u6f29': 3503, '\u874e': 3504, '\u7fb9': 3505, '\u6020': 3506, '\u75eb': 3507, '\u67da': 3508, '\u5f27': 3509, '\u71ee': 3510, '\u8eaf': 3511, '\u587e': 3512, '\u718f': 3513, '\u9504': 3514, '\u6c82': 3515, '\u4ea2': 3516, '\u6a71': 3517, '\u9776': 3518, '\u72b8': 3519, '\u6869': 3520, '\u4f0e': 3521, '\u6441': 3522, '\u7422': 3523, '\u8725': 3524, '\u8c1a': 3525, '\u94bc': 3526, '\u4fd1': 3527, '\u7a1a': 3528, '\u94ae': 3529, '\u9ae6': 3530, '\u73c8': 3531, '\u8611': 3532, '\u606c': 3533, '\u6e5b': 3534, '\u5406': 3535, '\u67b7': 3536, '\u7696': 3537, '\u72d2': 3538, '\u7f1a': 3539, '\u75de': 3540, '\u998f': 3541, '\u6d8e': 3542, '\u75ae': 3543, '\u98d2': 3544, '\u7825': 3545, '\u8317': 3546, '\u8bb9': 3547, '\u9cde': 3548, '\u60cb': 3549, '\u85fb': 3550, '\u9050': 3551, '\u54b3': 3552, '\u75fc': 3553, '\u76ce': 3554, '\u795f': 3555, '\u96bd': 3556, '\u57a6': 3557, '\u8892': 3558, '\u781d': 3559, '\u57e0': 3560, '\u997a': 3561, '~': 3562, '\u668c': 3563, '\u6ee4': 3564, '\u9074': 3565, '\u9a79': 3566, '\u7b77': 3567, '\u6d9d': 3568, '\u508d': 3569, '\u63ba': 3570, '\u4fa5': 3571, '\u3010': 3572, '\u3011': 3573, '\u996a': 3574, '\u7738': 3575, '\u82b7': 3576, '\u620e': 3577, '\u96c1': 3578, '\u5e90': 3579, '\u94ff': 3580, '\u68b5': 3581, '\u7b5b': 3582, '\u51f9': 3583, '\u6292': 3584, '\u7941': 3585, '\u53e8': 3586, '\u7430': 3587, '\u7316': 3588, '\u532e': 3589, '\u53f1': 3590, '\u59dd': 3591, '\u701b': 3592, '\u7736': 3593, '\u690e': 3594, '\u602f': 3595, '\u64a9': 3596, '\u5d3d': 3597, '\uff0b': 3598, '\u5ce8': 3599, '\u82bd': 3600, '\u51a2': 3601, '\u77a7': 3602, '\u9992': 3603, '\u6b86': 3604, '\u5b7d': 3605, '\u6ec7': 3606, '\u71a0': 3607, '\u53ae': 3608, '\u88d8': 3609, '\u5b62': 3610, '\u9535': 3611, '\u87d1': 3612, '\u9165': 3613, '\u94ee': 3614, '\u7405': 3615, '\u8c29': 3616, '\u72c8': 3617, '\u9a7c': 3618, '\u95fd': 3619, '\u80f3': 3620, '\u541f': 3621, '\u6dc6': 3622, '\u533e': 3623, '\u9ebe': 3624, '\u8e1e': 3625, '\u73c9': 3626, '\u6109': 3627, '\u83bd': 3628, '\u5321': 3629, '\u5bf0': 3630, '\uff0d': 3631, '\u5830': 3632, '\u8110': 3633, '\u5156': 3634, '\u83c0': 3635, '\u51c4': 3636, '\u545b': 3637, '\u5455': 3638, '\u7c41': 3639, '\u4f6f': 3640, '\u5462': 3641, '\u5b71': 3642, '\u5a6a': 3643, '\u60ee': 3644, '\u9aa5': 3645, '\u7315': 3646, '\u8baa': 3647, '\u7bb4': 3648, '\u72e1': 3649, '\u89d1': 3650, '\u77dc': 3651, '\u832c': 3652, '\u62c4': 3653, '\u6fd1': 3654, '\u8734': 3655, '\u5a76': 3656, '\u60ec': 3657, '\u4e15': 3658, '\u5bc5': 3659, '\u75b5': 3660, '\u51db': 3661, '\u8fe6': 3662, '\u916a': 3663, '\u6bcb': 3664, '\u824b': 3665, '\u75bd': 3666, '\u8c4c': 3667, '\u776b': 3668, '\u87b3': 3669, '\u4fe8': 3670, '\u6c93': 3671, '\u6829': 3672, '\u79e4': 3673, '\u6b7c': 3674, '\u54a4': 3675, '\u919b': 3676, '\u780c': 3677, '\u8912': 3678, '\u6f33': 3679, '\u7ff1': 3680, '\u778c': 3681, '\u5014': 3682, '\u6c1f': 3683, '\u783a': 3684, '\u6e0e': 3685, '\u541d': 3686, '\u9119': 3687, '\u7261': 3688, '\u8511': 3689, '\u67ff': 3690, '\u7f00': 3691, '\u8910': 3692, '\u5f1b': 3693, '\u8ba3': 3694, '\u663c': 3695, '\u8385': 3696, '\u8ff8': 3697, '\u9ed4': 3698, '\u68d8': 3699, '\u7c27': 3700, '\u55c5': 3701, '\u7b60': 3702, '\u732c': 3703, '\u568e': 3704, '\u961c': 3705, '\u6da4': 3706, '\u76b1': 3707, '\u7109': 3708, '\u64ae': 3709, '\u9997': 3710, '\u75b9': 3711, '\u96b6': 3712, '\u5450': 3713, '\u9ecf': 3714, '\u8bdf': 3715, '\u7459': 3716, '\u94a8': 3717, '\u9563': 3718, '\u7708': 3719, '\u6f2a': 3720, '\u9e66': 3721, '\u9e49': 3722, '\u5499': 3723, '\u9c87': 3724, '\u63a3': 3725, '\u5495': 3726, '\u65a1': 3727, '\u8682': 3728, '\u60f6': 3729, '\u5b17': 3730, '\u94e7': 3731, '\u797a': 3732, '\u7357': 3733, '\u561f': 3734, '\u52be': 3735, '\u7574': 3736, '\u8d58': 3737, '\u80da': 3738, '\u58a9': 3739, '\u8dbe': 3740, '\u54ab': 3741, '\u71ce': 3742, '\u96cc': 3743, '\u9540': 3744, '\u553e': 3745, '\u94b5': 3746, '\u6d31': 3747, '\u94ca': 3748, '\u852b': 3749, '\u8944': 3750, '\u5241': 3751, '\uff1e': 3752, '\u82c7': 3753, '\u864f': 3754, '\u776c': 3755, '\u8f7c': 3756, '\u8171': 3757, '\u759a': 3758, '\u7741': 3759, '\u67d1': 3760, '\u5742': 3761, '\u5942': 3762, '\u563b': 3763, '\u7791': 3764, '\u78fa': 3765, '\u50fb': 3766, '\u708a': 3767, '\u6221': 3768, '\u9640': 3769, '\u8c0c': 3770, '\u8747': 3771, '\u6eb6': 3772, '\u61f5': 3773, '\u56bc': 3774, '\u8bcb': 3775, '\u60b8': 3776, '\u68a7': 3777, '\u8dc6': 3778, '\u82a5': 3779, '\u5055': 3780, '\u5228': 3781, '\u667e': 3782, '\u69bb': 3783, '\u67a3': 3784, '\u7096': 3785, '\uff15': 3786, '\u7940': 3787, '\u5919': 3788, '\u6808': 3789, '\u5e27': 3790, '\u5c94': 3791, '\u7766': 3792, '\u70af': 3793, '\u94b2': 3794, '\u5eb9': 3795, '\u84c1': 3796, '\u8707': 3797, '\u816d': 3798, '\u9522': 3799, '\u2605': 3800, '\u733e': 3801, '\u7337': 3802, '\u8bff': 3803, '\u60e6': 3804, '\u6868': 3805, '\u777e': 3806, '\u9713': 3807, '\u54fa': 3808, '\u7435': 3809, '\u7436': 3810, '\u63e9': 3811, '\u752c': 3812, '\u6452': 3813, '\u630e': 3814, '\u55d2': 3815, '\u6963': 3816, '\uff2f': 3817, '\uff2d': 3818, '\u819b': 3819, '\u8815': 3820, '\u6421': 3821, '\u7f28': 3822, '\u6de4': 3823, '\u7600': 3824, '\u9492': 3825, '\u6854': 3826, '\u80f0': 3827, '\u80aa': 3828, '\u62a1': 3829, '\u9f3e': 3830, '\u5ffb': 3831, '\u6e0d': 3832, '\u79f8': 3833, '\u79c6': 3834, '\u680e': 3835, '\u9cd6': 3836, '\u55fd': 3837, '\u74a9': 3838, '\u828d': 3839, '\u7428': 3840, '\u7729': 3841, '\u5fff': 3842, '\u62ee': 3843, '\u6593': 3844, '\u9609': 3845, '\u895f': 3846, '\u9765': 3847, '\u85d0': 3848, '\u9edc': 3849, '\u7f61': 3850, '\u62ed': 3851, '\u6c5e': 3852, '\u67e0': 3853, '\u6aac': 3854, '\u9ab7': 3855, '\u9ac5': 3856, '\u9530': 3857, '\u6654': 3858, '\u6813': 3859, '\u8be9': 3860, '\u6a90': 3861, '\u949c': 3862, '^': 3863, '\u745a': 3864, '\u00d7': 3865, '\u57a3': 3866, '\u846b': 3867, '\u6f8e': 3868, '\u6c7e': 3869, '\u905b': 3870, '\u8db8': 3871, '\u9955': 3872, '\u992e': 3873, '\u59e3': 3874, '\u71e5': 3875, '\u62e3': 3876, '\u9570': 3877, '\u8473': 3878, '\u7bd3': 3879, '\u829c': 3880, '\u6930': 3881, '\u77a0': 3882, '\u7aa8': 3883, '\u8214': 3884, '\u6657': 3885, '\u7a23': 3886, '\u5ed3': 3887, '\u79ba': 3888, '\u81fc': 3889, '\u59a4': 3890, '\u7078': 3891, '\u8d3b': 3892, '\u2162': 3893, '\u85b0': 3894, '\u728a': 3895, '\u5e4c': 3896, '\u6f13': 3897, '\u5494': 3898, '\u87fe': 3899, '\u557c': 3900, '\u00b0': 3901, '\u83e0': 3902, '\u6376': 3903, '\u7099': 3904, '\u6726': 3905, '\u80e7': 3906, '\u68e3': 3907, '\u8c32': 3908, '\u577b': 3909, '\u998b': 3910, '\u7f04': 3911, '\u9cb2': 3912, '\u5c3b': 3913, '\u7d0a': 3914, '\u8be7': 3915, '\u604d': 3916, '\u55d1': 3917, '\u70d8': 3918, '\u8084': 3919, '\u4f91': 3920, '\u6a84': 3921, '\u5a32': 3922, '\u6994': 3923, '\u8e6c': 3924, '\u7f94': 3925, '\u7bf1': 3926, '\u78f7': 3927, '\u8e2f': 3928, '\u8e85': 3929, '\u6cfe': 3930, '\u61ca': 3931, '\u7cdc': 3932, '\u7b94': 3933, '\u5349': 3934, '\uff1b': 3935, '\u5412': 3936, '\u9ccd': 3937, '\u561b': 3938, '\u8038': 3939, '\u8c19': 3940, '\u86e4': 3941, '\u87c6': 3942, '\u822b': 3943, '\u61a8': 3944, '\u6cf8': 3945, '\u536f': 3946, '\u68f1': 3947, '\u9e35': 3948, '\u53fc': 3949, '\u64d8': 3950, '\u7b75': 3951, '\u8e42': 3952, '\u8e8f': 3953, '\u552c': 3954, '\u8238': 3955, '\u5a08': 3956, '\u65bc': 3957, '\u55f2': 3958, '\u762a': 3959, '\u7898': 3960, '\u9c88': 3961, '\u8364': 3962, '\uff21': 3963, '\u7c9f': 3964, '\u62c2': 3965, '\u5484': 3966, '\u59be': 3967, '\u951a': 3968, '\u9f85': 3969, '\u70ec': 3970, '\u8e52': 3971, '\u8dda': 3972, '\u5c27': 3973, '\u54c7': 3974, '\u6bd3': 3975, '\u60f0': 3976, '\u7cef': 3977, '\u6e43': 3978, '\u55b5': 3979, '\u7b71': 3980, '\u9ca4': 3981, '\u8dcb': 3982, '\u9e43': 3983, '\u8862': 3984, '\u837c': 3985, '\u6175': 3986, '\u94e0': 3987, '\u9e28': 3988, '\u6f88': 3989, '\uff1c': 3990, '\u54c6': 3991, '\u8c11': 3992, '\u6c22': 3993, '\u6eb1': 3994, '\u7281': 3995, '\u5d34': 3996, '\u6413': 3997, '\u79c3': 3998, '\u5636': 3999, '\u8146': 4000, '\u733f': 4001, '\u828a': 4002, '\u8c82': 4003, '\u75ff': 4004, '\u7747': 4005, '\u76c5': 4006, '\u7cb1': 4007, '\u6055': 4008, '\u7ead': 4009, '\u9525': 4010, '\u7eb0': 4011, '\u6002': 4012, '\u6c72': 4013, '\u69ad': 4014, '\u82df': 4015, '\u56f1': 4016, '\u870d': 4017, '\u87e0': 4018, '\u6c85': 4019, '\u77aa': 4020, '\uff10': 4021, '\u6840': 4022, '$': 4023, '\u8e74': 4024, '\u5e18': 4025, '\u5080': 4026, '\u5121': 4027, '\u61e6': 4028, '\u6cb1': 4029, '\u75d8': 4030, '\u54e6': 4031, '\u5b70': 4032, '\u7b19': 4033, '\u99a5': 4034, '\u620c': 4035, '\u6ee6': 4036, '\u81c6': 4037, '\u300c': 4038, '\u300d': 4039, '\u5ce4': 4040, '\u80ef': 4041, '\u8f84': 4042, '\u67a5': 4043, '\uff11': 4044, '\u6525': 4045, '\u908b': 4046, '\u9062': 4047, '\u6d9f': 4048, '\u70ca': 4049, '\u5a40': 4050, '\u76cf': 4051, '\u70e9': 4052, '\u75f9': 4053, '\u63f6': 4054, '\u63c4': 4055, '\u6ccc': 4056, '\u5352': 4057, '\u5f77': 4058, '\u5fa8': 4059, '\u956d': 4060, '\u9c7f': 4061, '\u54a3': 4062, '\u4f09': 4063, '\u8340': 4064, '\u98e8': 4065, '\u8517': 4066, '\u6e32': 4067, '\u94e8': 4068, '\u88c6': 4069, '\u7b06': 4070, '\u961a': 4071, '\u956f': 4072, '\u7b0b': 4073, '\u777d': 4074, '\uff0e': 4075, '\u62e7': 4076, '\u80b4': 4077, '\u6dc4': 4078, '\u8c27': 4079, '\u62f4': 4080, '\u7728': 4081, '\u7980': 4082, '\u6382': 4083, '\u7638': 4084, '\u8e69': 4085, '\u6e25': 4086, '\u69b7': 4087, '\u72f0': 4088, '\u72de': 4089, '\u9506': 4090, '\u761f': 4091, '\u8e4b': 4092, '\u7ee5': 4093, '\u9773': 4094, '\u6d43': 4095, '\u954c': 4096, '\u64b5': 4097, '\u8bd9': 4098, '\u6035': 4099, '\u9f2c': 4100, '\u5f0b': 4101, '\u7b20': 4102, '\uff29': 4103, '\u988a': 4104, '\u9647': 4105, '\u952d': 4106, '\u9534': 4107, '\u6c28': 4108, '\u7f2d': 4109, '\u83a0': 4110, '\u8093': 4111, '\u7f30': 4112, '\u65d6': 4113, '\u5480': 4114, '\u61cb': 4115, '\u60d8': 4116, '\u9cd7': 4117, '\u4fac': 4118, '\u9e5c': 4119, '\u9ab0': 4120, '\u75b1': 4121, '\u75e3': 4122, '\u75f0': 4123, '\u914c': 4124, '\u5cb1': 4125, '\u73c2': 4126, '\u556c': 4127, '\u8469': 4128, '\u817c': 4129, '\u8e0c': 4130, '\u8e87': 4131, '\u5d27': 4132, '\u9e29': 4133, '\u714a': 4134, '\u6538': 4135, '\u5f87': 4136, '\u821b': 4137, '\u85c9': 4138, '\u607f': 4139, '\u4e36': 4140, '\u5c09': 4141, '\u94db': 4142, '\u80ae': 4143, '\u94e4': 4144, '\u6e89': 4145, '\u97ec': 4146, '\u7f42': 4147, '\u6e2d': 4148, '\u94b3': 4149, '\uff14': 4150, '\uff2c': 4151, '\u520e': 4152, '\u8398': 4153, '\u9e6b': 4154, '\u6f8d': 4155, '\u954d': 4156, '\u82d4': 4157, '\u6f3e': 4158, '\u868c': 4159, '\u781a': 4160, '\u81c3': 4161, '\u00e9': 4162, '\u86a4': 4163, '\u9a9c': 4164, '\u759f': 4165, '\u710a': 4166, '\u7c38': 4167, '\u82d2': 4168, '\u8fab': 4169, '\u7b03': 4170, '\u6c50': 4171, '\u90af': 4172, '\u90f8': 4173, '\u8dec': 4174, '\u6edf': 4175, '\u6f31': 4176, '\u5018': 4177, '\u765c': 4178, '\u6684': 4179, '\u6f15': 4180, '\u51bc': 4181, '\u86af': 4182, '\u8693': 4183, '\u98d5': 4184, '\u7960': 4185, '\u830e': 4186, '\u7663': 4187, '\u2160': 4188, '\u94ba': 4189, '\u51c7': 4190, '\u90f4': 4191, '\u634e': 4192, '\u8572': 4193, '\u9f9b': 4194, '\u955d': 4195, '\u9190': 4196, '\u77d7': 4197, '\u618e': 4198, '\u59ca': 4199, '\u6414': 4200, '\u988c': 4201, '\u989a': 4202, '\u558b': 4203, '\uff19': 4204, '\u699b': 4205, '\u9068': 4206, '\u77a5': 4207, '\u631e': 4208, '\u705e': 4209, '\u4ff3': 4210, '\u94ec': 4211, '\u94f3': 4212, '\u5a67': 4213, '\u5729': 4214, '\u7f2e': 4215, '\u6c2a': 4216, '\u998a': 4217, '\u9611': 4218, '\u835e': 4219, '\u65ce': 4220, '\u8f72': 4221, '\u2165': 4222, '\u7949': 4223, '\u6636': 4224, '\u757f': 4225, '\u9537': 4226, '\u7ecc': 4227, '\u94d6': 4228, '\u9542': 4229, '\u6f78': 4230, '\u53f5': 4231, '\u9cc5': 4232, '\u86c0': 4233, '\u7fca': 4234, '\u9991': 4235, '\u94af': 4236, '\u86ac': 4237, '\u7762': 4238, '\u63ac': 4239, '\u9b03': 4240, '\u86c6': 4241, '\u80f1': 4242, '\u7bc6': 4243, '\uff26': 4244, '\u63c9': 4245, '\u949a': 4246, '\u951f': 4247, '\u9abc': 4248, '\u6482': 4249, '\u60da': 4250, '\u76c2': 4251, '\u2166': 4252, '\u500c': 4253, '\u91b4': 4254, '\u6d5c': 4255, '\u84df': 4256, '\u6d9e': 4257, '\u54fc': 4258, '\u51cb': 4259, '\u70f7': 4260, '\u89d0': 4261, '\u8708': 4262, '\u86a3': 4263, '\u560e': 4264, '\u98a6': 4265, '\u9cb6': 4266, '\uff06': 4267, '\u53a9': 4268, '\uff13': 4269, '\u6805': 4270, '\u542d': 4271, '\u3002': 4272, '\u2018': 4273, '\u2019': 4274, '\u768e': 4275, '\u25cb': 4276, '\u8654': 4277, '\u6da3': 4278, '\u67c1': 4279, '\u71b9': 4280, '\u7812': 4281, '\u5d82': 4282, '\u7a20': 4283, '\u9083': 4284, '\uff3f': 4285, '\u6004': 4286, '\u807f': 4287, '\u9e2b': 4288, '\u5e44': 4289, '\u7c07': 4290, '\u659f': 4291, '\u71a8': 4292, '\u753a': 4293, '\u6479': 4294, '\u8198': 4295, '\u03a3': 4296, '\uff27': 4297, '\uff34': 4298, '\u8869': 4299, '\u8859': 4300, '\u7bdd': 4301, '\u795a': 4302, '\u91ba': 4303, '\u8925': 4304, '\u54a7': 4305, '\u82d3': 4306, '\u9c91': 4307, '\u5555': 4308, '\u809b': 4309, '\u7699': 4310, '\u94e1': 4311, '\u690b': 4312, '\u6043': 4313, '\u6063': 4314, '\u8bb4': 4315, '\u7bab': 4316, '\u7934': 4317, '\u8151': 4318, '\u78b4': 4319, '\u814b': 4320, '\u6dcc': 4321, '\u502d': 4322, '\u7edb': 4323, '\u03b2': 4324, '\u6ce0': 4325, '\uff18': 4326, '\uff16': 4327, '\u4e98': 4328, '\u7599': 4329, '\u7629': 4330, '\u8386': 4331, '\u6410': 4332, '\u9981': 4333, '\u5600': 4334, '\u4ff8': 4335, '\u85e9': 4336, '\u79e3': 4337, '\u562d': 4338, '\u6666': 4339, '\u4f22': 4340, '\u5958': 4341, '\u81b3': 4342, '\u9c9f': 4343, '\u6249': 4344, '\u5364': 4345, '\u9977': 4346, '\u7a3c': 4347, '\u9730': 4348, '\u74e3': 4349, '\u6634': 4350, '\u75a3': 4351, '\u9708': 4352, '\u7fb8': 4353, '\u556a': 4354, '\u7830': 4355, '\u8d73': 4356, '\u94a0': 4357, '\u8568': 4358, '\u94bd': 4359, '\u8bcf': 4360, '\u8018': 4361, '\u5300': 4362, '\u60c6': 4363, '\u6005': 4364, '\u5556': 4365, '\u4edd': 4366, '\u7118': 4367, '\u6d95': 4368, '\u5594': 4369, '\u6fe1': 4370, '\u94e9': 4371, '\u8d4a': 4372, '\u55b1': 4373, '\u50ee': 4374, '\u9a85': 4375, '\u5514': 4376, '\u6ca5': 4377, '\u5490': 4378, '\u91c9': 4379, '\u9604': 4380, '\u9ecd': 4381, '\u740f': 4382, '\u8731': 4383, '\u853c': 4384, '\u748b': 4385, '\u60af': 4386, '\u4f2b': 4387, '\u2026': 4388, '\u57c2': 4389, '\u6371': 4390, '\u74e2': 4391, '\u5250': 4392, '\u9a9e': 4393, '\u523d': 4394, '\u59d8': 4395, '\u672d': 4396, '\u6b93': 4397, '\u5fe1': 4398, '\u67d2': 4399, '\u5eff': 4400, '\u83e9': 4401, '\u9e48': 4402, '\u622c': 4403, '\u75c9': 4404, '\u631b': 4405, '\u8757': 4406, '\u6db8': 4407, '\u8e59': 4408, '\u6dd6': 4409, '\u693f': 4410, '\u9511': 4411, '\u6308': 4412, '\u6ee2': 4413, '\u8366': 4414, '\u55dd': 4415, '\u675e': 4416, '\u9e38': 4417, '\u9e4b': 4418, '\u80f4': 4419, '\u5c98': 4420, '\u636d': 4421, '\u9616': 4422, '\u9e82': 4423, '\u7322': 4424, '\u72f2': 4425, '\u78ec': 4426, '\u598a': 4427, '\u5a20': 4428, '\u7178': 4429, '\u783e': 4430, '\u549a': 4431, '\u5106': 4432, '\u5d4b': 4433, '\u8671': 4434, '\uff23': 4435, '\uff33': 4436, '\u7ea3': 4437, '\u709d': 4438, '\u9532': 4439, '\u57a1': 4440, '\u84a1': 4441, '\u7350': 4442, '\u64c0': 4443, '\u7b90': 4444, '\u9554': 4445, '\u9131': 4446, '\u5155': 4447, '\u67e9': 4448, '\u837b': 4449, '\u7fbf': 4450, '\u542e': 4451, '\u94c6': 4452, '\u85ff': 4453, '\u6f66': 4454, '\u6a3d': 4455, '\u5e7a': 4456, '\u6ea5': 4457, '\u82aa': 4458, '\u874c': 4459, '\u86aa': 4460, '\u9a78': 4461, '\u891a': 4462, '\u9a88': 4463, '\u51bd': 4464, '\u5773': 4465, '\u5e1a': 4466, '\u606a': 4467, '\u5c49': 4468, '\u8936': 4469, '\u6dec': 4470, '\u5881': 4471, '\u79e7': 4472, '\u740a': 4473, '\u6f7c': 4474, '\u6cf1': 4475, '\u6901': 4476, '\u90ba': 4477, '\u772d': 4478, '\u789c': 4479, '\u8c36': 4480, '\u816e': 4481, '\u50ed': 4482, '\u938f': 4483, '\u4ffa': 4484, '\u5676': 4485, '\u568f': 4486, '\u5e96': 4487, '\u5ad8': 4488, '\u5b7a': 4489, '\u7620': 4490, '\u858f': 4491, '\u9f8c': 4492, '\u9f8a': 4493, '\u9efe': 4494, '\u90eb': 4495, '\u9642': 4496, '\u63e0': 4497, '\u84d6': 4498, '\u86fe': 4499, '\u6954': 4500, '\u7266': 4501, '\u6248': 4502, '\u7827': 4503, '\u53fd': 4504, '\u7b1e': 4505, '\u6026': 4506, '\u83e1': 4507, '\u81a6': 4508, '\u58d1': 4509, '\u765e': 4510, '\u55e6': 4511, '\u6123': 4512, '\u94f5': 4513, '\u8c18': 4514, '\u54cf': 4515, '\u86a9': 4516, '\u51fc': 4517, '\u50a7': 4518, '\u63d6': 4519, '\u54df': 4520, '\u741a': 4521, '\u6d04': 4522, '\u9088': 4523, '\u75a1': 4524, '\u916e': 4525, '\u55d4': 4526, '\u684e': 4527, '\u688f': 4528, '\u94c8': 4529, '\u9567': 4530, '\u5a23': 4531, '\u9701': 4532, '\u9e3e': 4533, '\u8be4': 4534, '\u919a': 4535, '\u4f43': 4536, '\u873f': 4537, '\u8712': 4538, '\u7800': 4539, '\u74e4': 4540, '\u6485': 4541, '\u90c4': 4542, '\u6d4a': 4543, '\u58c5': 4544, '\u94b4': 4545, '\u525c': 4546, '\u8153': 4547, '\u80eb': 4548, '\\x7f': 4549, '\u773a': 4550, '\u85c1': 4551, '\u4edf': 4552, '\u5a75': 4553, '\u95f1': 4554, '\u4e28': 4555, '\u58d5': 4556, '\u86ce': 4557, '\u5bd0': 4558, '\u75e7': 4559, '\u97ed': 4560, '\u812f': 4561, '\u56b7': 4562, '\u5eb5': 4563, '\u6a7c': 4564, '\u7ebe': 4565, '\u59af': 4566, '\u5a0c': 4567, '\u82de': 4568, '\u579b': 4569, '\u836d': 4570, '\u68a2': 4571, '\u948e': 4572, '\u80ed': 4573, '\u9b51': 4574, '\u87a8': 4575, '\u8888': 4576, '\u88df': 4577, '\u8e49': 4578, '\u8dce': 4579, '\u9e22': 4580, '\u9051': 4581, '\u7cb3': 4582, '\u8d2e': 4583, '\u73e5': 4584, '\u6e4d': 4585, '\u846d': 4586, '\u52ad': 4587, '\u953a': 4588, '\u8210': 4589, '\u94f1': 4590, '\u7852': 4591, '\u94e3': 4592, '\u2248': 4593, '\u8365': 4594, '\u550f': 4595, '\u853a': 4596, '\u6b84': 4597, '\u996c': 4598, '\u88f3': 4599, '\u501c': 4600, '\u50a5': 4601, '\u4fda': 4602, '\u975b': 4603, '\u5c7a': 4604, '\u835a': 4605, '\u5cbf': 4606, '\u5d99': 4607, '\u5ccb': 4608, '\u9035': 4609, '\u9e8b': 4610, '\u853b': 4611, '\u6f62': 4612, '\u55e3': 4613, '\u7e82': 4614, '\u539d': 4615, '\u94e2': 4616, '\u8fc2': 4617, '\u9cab': 4618, '\u9672': 4619, '\u960b': 4620, '\u7284': 4621, '\u2260': 4622, '\u6ec1': 4623, '\u9149': 4624, '\u4f7b': 4625, '\u5b34': 4626, '\u7688': 4627, '\u6862': 4628, '\u8e76': 4629, '\u64e4': 4630, '\u8fe4': 4631, '\u9026': 4632, '\u8244': 4633, '\u83ba': 4634, '\u6115': 4635, '\u834f': 4636, '\u5a7a': 4637, '\u692d': 4638, '\u6cf7': 4639, '\u9cbc': 4640, '\u789a': 4641, '\u80e5': 4642, '\u95f0': 4643, '\u65cc': 4644, '\u5c96': 4645, '\u72e8': 4646, '\u62d7': 4647, '\u7837': 4648, '\u524c': 4649, '\u7a37': 4650, '\u5197': 4651, '\u98a7': 4652, '\u7b95': 4653, '\u586c': 4654, '\u75d4': 4655, '\u9c85': 4656, '\u7850': 4657, '\u83d8': 4658, '\u836a': 4659, '\u6d54': 4660, '\u62c8': 4661, '\u5811': 4662, '\u6f2f': 4663, '\u9889': 4664, '\u77b3': 4665, '\u8d45': 4666, '\u9aa1': 4667, '\u6daa': 4668, '\u8e2e': 4669, '\u9a6e': 4670, '\u221a': 4671, '\u6ceb': 4672, '\u7ce8': 4673, '\u565c': 4674, '\u64b7': 4675, '\u73d0': 4676, '\uff12': 4677, '\u909b': 4678, '\u5d03': 4679, '\u7bd9': 4680, '\u6dde': 4681, '\u998d': 4682, '\u5e61': 4683, '\u6c1a': 4684, '\u2030': 4685, '\u96f9': 4686, '\u9acb': 4687, '\u9cb1': 4688, '\u4f57': 4689, '\u752d': 4690, '\u6ea7': 4691, '\u6cde': 4692, '\u9a90': 4693, '\u5639': 4694, '\u547b': 4695, '\u55bd': 4696, '\u9041': 4697, '\u7ee2': 4698, '\u77ec': 4699, '\u5d02': 4700, '\u6e6e': 4701, '\u5eb6': 4702, '\u783b': 4703, '\u9f2f': 4704, '\uff32': 4705, '\uff22': 4706, '\u612b': 4707, '\uff30': 4708, '\u75e2': 4709, '\u6e8f': 4710, '\u9984': 4711, '\u9968': 4712, '\u796f': 4713, '\u535e': 4714, '\u9974': 4715, '\u82f7': 4716, '\u8ddb': 4717, '\u9cd0': 4718, '\u8884': 4719, '\u64de': 4720, '\u8c89': 4721, '\u8288': 4722, '\u5657': 4723, '\u6396': 4724, '\u9517': 4725, '\u8014': 4726, '\u2016': 4727, '\u6e4e': 4728, '\u83cf': 4729, '\u7f25': 4730, '\u7f08': 4731, '\u75b4': 4732, '\u6dc5': 4733, '\u4ede': 4734, '\u53df': 4735, '\u5ad4': 4736, '\u6a28': 4737, '\u606b': 4738, '\u8be3': 4739, '\u53c1': 4740, '\u6c2e': 4741, '\u66f3': 4742, '\u8191': 4743, '\u5ce6': 4744, '\u652b': 4745, '\u9e44': 4746, '\u5544': 4747, '\u61a9': 4748, '\u9791': 4749, '\u57a0': 4750, '\u9e55': 4751, '\u911e': 4752, '\u5478': 4753, '\uff36': 4754, '\u73b7': 4755, '\u7601': 4756, '\u86b1': 4757, '\u00a7': 4758, '\u970e': 4759, '<UNK>': 4760, '<PAD>': 4761}\nVocab size: 4762\n\n14002it [00:00, 46239.08it\/s]119988it [00:03, 37652.90it\/s]\n\nTime usage: 3.715178966522217\nStart Training...\n\nepoch = 1 :  train_loss = 0.710, train_acc = 50.87%, dev_loss = 0.686, dev_acc = 53.61%\nepoch = 2 :  train_loss = 0.693, train_acc = 52.07%, dev_loss = 0.695, dev_acc = 50.37%\nepoch = 3 :  train_loss = 0.690, train_acc = 52.64%, dev_loss = 0.703, dev_acc = 52.38%\nepoch = 4 :  train_loss = 0.690, train_acc = 52.29%, dev_loss = 0.696, dev_acc = 52.15%\nepoch = 5 :  train_loss = 0.688, train_acc = 52.53%, dev_loss = 0.704, dev_acc = 52.01%\nepoch = 6 :  train_loss = 0.684, train_acc = 52.87%, dev_loss = 0.700, dev_acc = 52.65%\nepoch = 7 :  train_loss = 0.685, train_acc = 53.21%, dev_loss = 0.694, dev_acc = 53.20%\nepoch = 8 :  train_loss = 0.681, train_acc = 53.68%, dev_loss = 0.711, dev_acc = 51.15%\nepoch = 9 :  train_loss = 0.679, train_acc = 53.93%, dev_loss = 0.705, dev_acc = 52.41%\nepoch = 10 :  train_loss = 0.676, train_acc = 54.22%, dev_loss = 0.705, dev_acc = 50.55%\nepoch = 11 :  train_loss = 0.676, train_acc = 54.47%, dev_loss = 0.725, dev_acc = 52.14%\nepoch = 12 :  train_loss = 0.677, train_acc = 53.82%, dev_loss = 0.703, dev_acc = 51.69%\nepoch = 13 :  train_loss = 0.663, train_acc = 57.83%, dev_loss = 0.706, dev_acc = 52.45%\nepoch = 14 :  train_loss = 0.673, train_acc = 54.70%, dev_loss = 0.706, dev_acc = 51.09%\nepoch = 15 :  train_loss = 0.664, train_acc = 57.07%, dev_loss = 0.679, dev_acc = 62.89%\nepoch = 16 :  train_loss = 0.674, train_acc = 54.61%, dev_loss = 0.722, dev_acc = 50.65%\nepoch = 17 :  train_loss = 0.666, train_acc = 56.64%, dev_loss = 0.699, dev_acc = 59.89%\nepoch = 18 :  train_loss = 0.665, train_acc = 56.30%, dev_loss = 0.715, dev_acc = 50.62%\nepoch = 19 :  train_loss = 0.674, train_acc = 54.50%, dev_loss = 0.708, dev_acc = 53.67%\nepoch = 20 :  train_loss = 0.670, train_acc = 55.22%, dev_loss = 0.718, dev_acc = 53.39%\nepoch = 21 :  train_loss = 0.671, train_acc = 54.64%, dev_loss = 0.714, dev_acc = 52.66%\nepoch = 22 :  train_loss = 0.664, train_acc = 56.20%, dev_loss = 0.716, dev_acc = 52.83%\nepoch = 23 :  train_loss = 0.656, train_acc = 58.69%, dev_loss = 0.706, dev_acc = 55.55%\nepoch = 24 :  train_loss = 0.663, train_acc = 57.57%, dev_loss = 0.703, dev_acc = 56.21%\nepoch = 25 :  train_loss = 0.666, train_acc = 56.45%, dev_loss = 0.709, dev_acc = 51.55%\nepoch = 26 :  train_loss = 0.662, train_acc = 56.77%, dev_loss = 0.705, dev_acc = 52.80%\nepoch = 27 :  train_loss = 0.650, train_acc = 60.24%, dev_loss = 0.709, dev_acc = 53.09%\nepoch = 28 :  train_loss = 0.663, train_acc = 55.89%, dev_loss = 0.712, dev_acc = 54.12%\nepoch = 29 :  train_loss = 0.662, train_acc = 55.80%, dev_loss = 0.713, dev_acc = 51.41%\nepoch = 30 :  train_loss = 0.659, train_acc = 58.09%, dev_loss = 0.709, dev_acc = 52.50%\nepoch = 31 :  train_loss = 0.672, train_acc = 54.47%, dev_loss = 0.719, dev_acc = 52.73%\nepoch = 32 :  train_loss = 0.668, train_acc = 55.06%, dev_loss = 0.717, dev_acc = 53.13%\nepoch = 33 :  train_loss = 0.665, train_acc = 55.47%, dev_loss = 0.713, dev_acc = 52.45%\nepoch = 34 :  train_loss = 0.663, train_acc = 55.80%, dev_loss = 0.713, dev_acc = 53.17%\nepoch = 35 :  train_loss = 0.663, train_acc = 55.46%, dev_loss = 0.713, dev_acc = 52.65%\nepoch = 36 :  train_loss = 0.649, train_acc = 60.63%, dev_loss = 0.688, dev_acc = 62.12%\nepoch = 37 :  train_loss = 0.640, train_acc = 63.75%, dev_loss = 0.689, dev_acc = 60.96%\nepoch = 38 :  train_loss = 0.652, train_acc = 59.89%, dev_loss = 0.697, dev_acc = 57.65%\nepoch = 39 :  train_loss = 0.655, train_acc = 58.81%, dev_loss = 0.706, dev_acc = 53.71%\nepoch = 40 :  train_loss = 0.661, train_acc = 55.94%, dev_loss = 0.707, dev_acc = 54.12%\nepoch = 41 :  train_loss = 0.654, train_acc = 57.61%, dev_loss = 0.732, dev_acc = 53.49%\nepoch = 42 :  train_loss = 0.657, train_acc = 56.41%, dev_loss = 0.710, dev_acc = 54.98%\nepoch = 43 :  train_loss = 0.641, train_acc = 62.07%, dev_loss = 0.686, dev_acc = 62.00%\nepoch = 44 :  train_loss = 0.640, train_acc = 61.98%, dev_loss = 0.682, dev_acc = 61.96%\nepoch = 45 :  train_loss = 0.635, train_acc = 62.63%, dev_loss = 0.701, dev_acc = 56.87%\nepoch = 46 :  train_loss = 0.658, train_acc = 56.43%, dev_loss = 0.700, dev_acc = 59.63%\nepoch = 47 :  train_loss = 0.634, train_acc = 63.87%, dev_loss = 0.692, dev_acc = 61.46%\nepoch = 48 :  train_loss = 0.646, train_acc = 60.44%, dev_loss = 0.706, dev_acc = 57.70%\nepoch = 49 :  train_loss = 0.653, train_acc = 57.99%, dev_loss = 0.716, dev_acc = 52.55%\nepoch = 50 :  train_loss = 0.655, train_acc = 57.76%, dev_loss = 0.719, dev_acc = 52.46%\n================================================================================================================================\ntest_loss: 0.675      test_acc: 62.90%\nend<\/code><\/pre>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913212914960.png\" style=\"height:300px\">\n<\/p>\n<p align=\"center\">\n  <img decoding=\"async\" src=\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240913212931156.png\" style=\"height:300px\">\n<\/p>\n<h2><img decoding=\"async\" src=\"https:\/\/img.icons8.com\/dusk\/64\/000000\/prize.png\" style=\"height:50px;display:inline\"> Credits<\/h2>\n<hr \/>\n<ul>\n<li>Icons made by <a href=\"https:\/\/www.flaticon.com\/authors\/becris\" title=\"Becris\">Becris<\/a> from <a href=\"https:\/\/www.flaticon.com\/\" title=\"Flaticon\">www.flaticon.com<\/a><\/li>\n<li>Icons from <a href=\"https:\/\/icons8.com\/\">Icons8.com<\/a> - <a href=\"https:\/\/icons8.com\">https:\/\/icons8.com<\/a><\/li>\n<li><a href=\"https:\/\/d2l.ai\/chapter_recurrent-neural-networks\/index.html\">Dive Into Deep Learning - Recurrent Neural Networks<\/a><\/li>\n<li><a href=\"https:\/\/atcold.github.io\/pytorch-Deep-Learning\/en\/week12\/12-1\/\">DS-GA 1008 - NYU CENTER FOR DATA SCIENCE - Deep Sequence Modeling<\/a><\/li>\n<li><a href=\"https:\/\/pytorch.org\/tutorials\/beginner\/text_sentiment_ngrams_tutorial.html\">Text classification with the torchtext library<br \/>\n<\/a><\/li>\n<li><a href=\"https:\/\/www.borealisai.com\/research-blogs\/tutorial-17-transformers-iii-training\/\">Tricks For Training Transformers - Borealis AI - P. Xu, S. Prince<\/a><\/li>\n<li><a href=\"https:\/\/taldatech.github.io\">Tal Daniel<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Deep Learning create by Arwin Yu Tutorial 03 &#8211; Recurren [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1893,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,24],"tags":[19],"class_list":["post-1878","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-18","category-24","tag-19"],"_links":{"self":[{"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/posts\/1878","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.gnn.club\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1878"}],"version-history":[{"count":8,"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/posts\/1878\/revisions"}],"predecessor-version":[{"id":1973,"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/posts\/1878\/revisions\/1973"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.gnn.club\/index.php?rest_route=\/wp\/v2\/media\/1893"}],"wp:attachment":[{"href":"http:\/\/www.gnn.club\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1878"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.gnn.club\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1878"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.gnn.club\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1878"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}