initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,156 @@
+---
+license: apache-2.0
+---
+# Metom (めとん)
+
+The **Metom** is a Vision Transformer (ViT) based **Kuzushiji** classifier.
+The model takes an image with one character and returns what the character is.
+**This model is not an official SakanaAI product and is for research / educational purposes only.**
+
+**めとん**は Vision Transformer (ViT) ベースの**くずし字**分類器です。
+モデルは1文字が写った画像を受け取り、その文字がどの文字であるかを返します。
+**本モデルはSakanaAIの公式製品ではありません。研究・教育目的のみに利用してください。**
+
+*Japanese section follows English section (日本語セクションは英語セクションの後に続きます。)*
+
+--------------------------------------------------------------------------------
+
+This model was trained by using [日本古典籍くずし字データセット](http://codh.rois.ac.jp/char-shape/book/).
+This dataset contains 1,086,326 characters in 4,328 types of Kuzushiji.
+However, we used only 2,703 types of characters that appeared at least 5 times in the dataset.
+
+The dataset was split into train, validation, and test subsets in a ratio of 3:1:1.
+As a result, the train subset contained 649,932 characters, the validation subset contained 216,644 characters, and the test subset contained 216,645 characters.
+
+The model was trained on the train subset, and hyperparameters were tuned based on the performance on the validation subset.
+The final evaluation on the test subset yielded a micro accuracy of 0.9722 and a macro accuracy of 0.8354.
+
+## Usage
+Please see also [Google Colab Notebook](https://colab.research.google.com/drive/1jFMZENoTjjum3qlBxV0Q5dTxmpCvqlpf?usp=sharing).
+1. Install dependencies (Not required on Google Colab)
+```sh
+python -m pip install einops torch torchvision transformers
+
+# Optional (This is also required on Google Colab if you want to use FlashAttention-2)
+pip install flash-attn --no-build-isolation
+```
+
+2. Run the following code
+```python
+from io import BytesIO
+
+from PIL import Image
+import requests
+import torch
+from transformers import AutoModel, AutoProcessor
+
+repo_name = "SakanaAI/Metom"
+device = "cuda"
+torch_dtype = torch.float32  # This can also set `torch.float16` or `torch.bfloat16`
+
+def get_image(image_url: str) -> Image.Image:
+    return Image.open(BytesIO(requests.get(image_url).content)).convert("RGB")
+
+processor = AutoProcessor.from_pretrained(repo_name, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    repo_name,
+    torch_dtype=torch_dtype,
+    _attn_implementation="eager",  # This can also set `"sdpa"` or `"flash_attention_2"`
+    trust_remote_code=True
+).to(device=device)
+
+image1 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example1_4E00.jpg")  # An example image
+image_array1 = processor(images=image1, return_tensors="pt")["pixel_values"].to(device=device, dtype=torch_dtype)
+with torch.inference_mode():
+    print(model.get_predictions(image_array1))  # Returns the prediction label
+# ['一']
+
+image2 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example2_5B9A.jpg")  # An example image
+image3 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example3_5009.jpg")  # An example image
+image_array2 = processor(images=[image2, image3], return_tensors="pt")["pixel_values"].to(device=device, dtype=torch_dtype)
+with torch.inference_mode():
+    print(model.get_topk_labels(image_array2))  # Returns top-k prediction labels (label only)
+    # [['定', '芝', '乏', '淀', '実'], ['倉', '衾', '斜', '会', '急']]
+    print(model.get_topk_labels(image_array2, k=3, return_probs=True))  # Returns prediction top-k labels (label with probability)
+    # [[('定', 0.9979104399681091), ('芝', 0.0002953427319880575), ('乏', 0.00012814522779081017)], [('倉', 0.9862521290779114), ('衾', 0.0005956474924460053), ('斜', 0.00039981433656066656)]]
+```
+
+## Citation
+```bibtex
+@misc{Metom,
+    url    = {[https://huggingface.co/SakanaAI/Metom](https://huggingface.co/SakanaAI/Metom)},
+    title  = {Metom},
+    author = {Imajuku, Yuki and Clanuwat, Tarin}
+}
+```
+
+--------------------------------------------------------------------------------
+
+本モデルは[日本古典籍くずし字データセット](http://codh.rois.ac.jp/char-shape/book/)を用いて訓練されました。
+このデータセットは4,328種1,086,326枚のくずし字画像が含まれています。
+ですが、データセット中に最低5回以上出現する2,703種類の文字のみを利用しました。
+
+データセットは訓練、検証、テストの3つのセットに、比率が3:1:1となるように分割されました。
+その結果、訓練セットは649,932枚、検証セットは216,644枚、テストセットは216,645枚、画像が含まれました。
+
+本モデルは訓練セッ��のみを用いて学習され、検証セットにおける性能を見ながらハイパーパラメータを調整しました。
+最終的にテストセットにおける評価の結果、216,645枚全体の正解率は0.9722となり、2,703種類のクラス別正解率の平均は0.8354となりました。
+
+## 使用方法
+[Google Colab Notebook](https://colab.research.google.com/drive/1jFMZENoTjjum3qlBxV0Q5dTxmpCvqlpf?usp=sharing)もご確認ください。
+1. 依存ライブラリをインストールする (Google Colabを使う場合は不要)
+```sh
+python -m pip install einops torch torchvision transformers
+
+# 任意 (FlashAttention-2を使いたい場合はGoogle Colabを使う時でも必要)
+pip install flash-attn --no-build-isolation
+```
+
+2. 以下のコードを実行する
+```python
+from io import BytesIO
+
+from PIL import Image
+import requests
+import torch
+from transformers import AutoModel, AutoProcessor
+
+repo_name = "SakanaAI/Metom"
+device = "cuda"
+torch_dtype = torch.float32  # `torch.float16` や `torch.bfloat16` も指定可能
+
+def get_image(image_url: str) -> Image.Image:
+    return Image.open(BytesIO(requests.get(image_url).content)).convert("RGB")
+
+processor = AutoProcessor.from_pretrained(repo_name, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    repo_name,
+    torch_dtype=torch_dtype,
+    _attn_implementation="eager",  # `"sdpa"` や `"flash_attention_2"` も指定可能
+    trust_remote_code=True
+).to(device=device)
+
+image1 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example1_4E00.jpg")  # 画像例
+image_array1 = processor(images=image1, return_tensors="pt")["pixel_values"].to(device=device, dtype=torch_dtype)
+with torch.inference_mode():
+    print(model.get_predictions(image_array1))  # 予測ラベルを返す
+# ['一']
+
+image2 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example2_5B9A.jpg")  # 画像例
+image3 = get_image("https://huggingface.co/SakanaAI/Metom/resolve/main/examples/example3_5009.jpg")  # 画像例
+image_array2 = processor(images=[image2, image3], return_tensors="pt")["pixel_values"].to(device=device, dtype=torch_dtype)
+with torch.inference_mode():
+    print(model.get_topk_labels(image_array2))  # 上位k件の予測ラベルを返す (ラベルのみ)
+    # [['定', '芝', '乏', '淀', '実'], ['倉', '衾', '斜', '会', '急']]
+    print(model.get_topk_labels(image_array2, k=3, return_probs=True))  # 上位k件の予測ラベルを返す (ラベルと確率)
+    # [[('定', 0.9979104399681091), ('芝', 0.0002953427319880575), ('乏', 0.00012814522779081017)], [('倉', 0.9862521290779114), ('衾', 0.0005956474924460053), ('斜', 0.00039981433656066656)]]
+```
+
+## 引用
+```bibtex
+@misc{Metom,
+    url    = {[https://huggingface.co/SakanaAI/Metom](https://huggingface.co/SakanaAI/Metom)},
+    title  = {Metom},
+    author = {Imajuku, Yuki and Clanuwat, Tarin}
+}
+```

config.json

@@ -0,0 +1,2724 @@
+{
+  "auto_map": {
+    "AutoConfig": "configuration_metom.MetomConfig",
+    "AutoModel": "modeling_metom.MetomModel"
+  },
+  "channels": 3,
+  "depth": 6,
+  "dim": 256,
+  "dim_head": 32,
+  "dropout": 0.1,
+  "emb_dropout": 0.1,
+  "heads": 8,
+  "image_size": 128,
+  "labels": [
+    "\ud840\udc0b",
+    "\ud843\udd45",
+    "\u25a0",
+    "\u25b2",
+    "\u25b3",
+    "\u25cb",
+    "\u25cf",
+    "\ud85d\udf52",
+    "\u3002",
+    "\u3005",
+    "\u3006",
+    "\u3007",
+    "\u300c",
+    "\u300d",
+    "\u3031",
+    "\u3032",
+    "\u303b",
+    "\u303d",
+    "\u3042",
+    "\u3044",
+    "\u3046",
+    "\u3048",
+    "\u304a",
+    "\u304b",
+    "\u304c",
+    "\u304d",
+    "\u304e",
+    "\u304f",
+    "\u3050",
+    "\u3051",
+    "\u3052",
+    "\u3053",
+    "\u3054",
+    "\u3055",
+    "\u3056",
+    "\u3057",
+    "\u3058",
+    "\u3059",
+    "\u305a",
+    "\u305b",
+    "\u305c",
+    "\u305d",
+    "\u305e",
+    "\u305f",
+    "\u3060",
+    "\u3061",
+    "\u3062",
+    "\u3063",
+    "\u3064",
+    "\u3065",
+    "\u3066",
+    "\u3067",
+    "\u3068",
+    "\u3069",
+    "\u306a",
+    "\u306b",
+    "\u306c",
+    "\u306d",
+    "\u306e",
+    "\u306f",
+    "\u3070",
+    "\u3071",
+    "\u3072",
+    "\u3073",
+    "\u3074",
+    "\u3075",
+    "\u3076",
+    "\u3077",
+    "\u3078",
+    "\u3079",
+    "\u307a",
+    "\u307b",
+    "\u307c",
+    "\u307d",
+    "\u307e",
+    "\u307f",
+    "\u3080",
+    "\u3081",
+    "\u3082",
+    "\u3083",
+    "\u3084",
+    "\u3085",
+    "\u3086",
+    "\u3088",
+    "\u3089",
+    "\u308a",
+    "\u308b",
+    "\u308c",
+    "\u308d",
+    "\u308f",
+    "\u3090",
+    "\u3091",
+    "\u3092",
+    "\u3093",
+    "\u309d",
+    "\u309e",
+    "\u309f",
+    "\u30a1",
+    "\u30a2",
+    "\u30a4",
+    "\u30a6",
+    "\u30a8",
+    "\u30ab",
+    "\u30ac",
+    "\u30ad",
+    "\u30ae",
+    "\u30af",
+    "\u30b1",
+    "\u30b2",
+    "\u30b3",
+    "\u30b5",
+    "\u30b7",
+    "\u30b9",
+    "\u30ba",
+    "\u30bb",
+    "\u30bc",
+    "\u30bd",
+    "\u30bf",
+    "\u30c0",
+    "\u30c1",
+    "\u30c3",
+    "\u30c4",
+    "\u30c5",
+    "\u30c6",
+    "\u30c7",
+    "\u30c8",
+    "\u30c9",
+    "\u30ca",
+    "\u30cb",
+    "\u30cd",
+    "\u30ce",
+    "\u30cf",
+    "\u30d0",
+    "\u30d2",
+    "\u30d5",
+    "\u30d8",
+    "\u30d9",
+    "\u30db",
+    "\u30de",
+    "\u30df",
+    "\u30e0",
+    "\u30e1",
+    "\u30e2",
+    "\u30e4",
+    "\u30e8",
+    "\u30e9",
+    "\u30ea",
+    "\u30eb",
+    "\u30ec",
+    "\u30ed",
+    "\u30ef",
+    "\u30f1",
+    "\u30f2",
+    "\u30f3",
+    "\u30f6",
+    "\u30fb",
+    "\u30fd",
+    "\u30fe",
+    "\u3280",
+    "\u3281",
+    "\u3282",
+    "\u3283",
+    "\u3402",
+    "\u34b5",
+    "\u34db",
+    "\u351f",
+    "\u352b",
+    "\u355d",
+    "\u3761",
+    "\u4453",
+    "\u4e00",
+    "\u4e01",
+    "\u4e03",
+    "\u4e07",
+    "\u4e08",
+    "\u4e09",
+    "\u4e0a",
+    "\u4e0b",
+    "\u4e0d",
+    "\u4e0e",
+    "\u4e11",
+    "\u4e14",
+    "\u4e16",
+    "\u4e18",
+    "\u4e1e",
+    "\u4e21",
+    "\u4e26",
+    "\u4e2d",
+    "\u4e32",
+    "\u4e38",
+    "\u4e39",
+    "\u4e3b",
+    "\u4e3c",
+    "\u4e43",
+    "\u4e45",
+    "\u4e4b",
+    "\u4e4e",
+    "\u4e4f",
+    "\u4e57",
+    "\u4e59",
+    "\u4e5d",
+    "\u4e5e",
+    "\u4e5f",
+    "\u4e71",
+    "\u4e73",
+    "\u4e7e",
+    "\u4e80",
+    "\u4e86",
+    "\u4e88",
+    "\u4e89",
+    "\u4e8b",
+    "\u4e8c",
+    "\u4e8e",
+    "\u4e91",
+    "\u4e92",
+    "\u4e94",
+    "\u4e95",
+    "\u4e9b",
+    "\u4ea1",
+    "\u4ea4",
+    "\u4ea5",
+    "\u4ea6",
+    "\u4eab",
+    "\u4eac",
+    "\u4ead",
+    "\u4eba",
+    "\u4ec1",
+    "\u4ec7",
+    "\u4eca",
+    "\u4ecb",
+    "\u4ecd",
+    "\u4ecf",
+    "\u4ed4",
+    "\u4ed5",
+    "\u4ed6",
+    "\u4ed8",
+    "\u4ed9",
+    "\u4ee3",
+    "\u4ee4",
+    "\u4ee5",
+    "\u4eee",
+    "\u4ef0",
+    "\u4ef2",
+    "\u4ef6",
+    "\u4efb",
+    "\u4f0a",
+    "\u4f0f",
+    "\u4f10",
+    "\u4f11",
+    "\u4f1a",
+    "\u4f1d",
+    "\u4f2f",
+    "\u4f34",
+    "\u4f38",
+    "\u4f3a",
+    "\u4f3c",
+    "\u4f3d",
+    "\u4f46",
+    "\u4f4d",
+    "\u4f4e",
+    "\u4f4f",
+    "\u4f50",
+    "\u4f53",
+    "\u4f55",
+    "\u4f59",
+    "\u4f5c",
+    "\u4f73",
+    "\u4f75",
+    "\u4f76",
+    "\u4f7f",
+    "\u4f8b",
+    "\u4f8d",
+    "\u4f98",
+    "\u4f9b",
+    "\u4f9d",
+    "\u4fa0",
+    "\u4fad",
+    "\u4faf",
+    "\u4fbf",
+    "\u4fc2",
+    "\u4fc3",
+    "\u4fc4",
+    "\u4fca",
+    "\u4fce",
+    "\u4fd7",
+    "\u4fdd",
+    "\u4fdf",
+    "\u4fe1",
+    "\u4fe4",
+    "\u4fee",
+    "\u4fef",
+    "\u4ff3",
+    "\u4ff5",
+    "\u5009",
+    "\u500b",
+    "\u500d",
+    "\u5012",
+    "\u5016",
+    "\u5019",
+    "\u501a",
+    "\u501f",
+    "\u5026",
+    "\u502b",
+    "\u502d",
+    "\u5036",
+    "\u504f",
+    "\u5056",
+    "\u5065",
+    "\u5074",
+    "\u5076",
+    "\u5078",
+    "\u507d",
+    "\u5085",
+    "\u508d",
+    "\u5098",
+    "\u5099",
+    "\u50ac",
+    "\u50b7",
+    "\u50be",
+    "\u50c5",
+    "\u50c9",
+    "\u50cd",
+    "\u50cf",
+    "\u50d5",
+    "\u50e5",
+    "\u50e7",
+    "\u50f9",
+    "\u5100",
+    "\u5112",
+    "\u512a",
+    "\u5143",
+    "\u5144",
+    "\u5148",
+    "\u5149",
+    "\u514d",
+    "\u514e",
+    "\u5150",
+    "\u515c",
+    "\u5165",
+    "\u5168",
+    "\u516b",
+    "\u516c",
+    "\u516d",
+    "\u516e",
+    "\u5171",
+    "\u5175",
+    "\u5176",
+    "\u5177",
+    "\u5178",
+    "\u517c",
+    "\u5185",
+    "\u5186",
+    "\u518a",
+    "\u518d",
+    "\u5199",
+    "\u51a0",
+    "\u51a4",
+    "\u51a5",
+    "\u51ac",
+    "\u51b6",
+    "\u51b7",
+    "\u51c4",
+    "\u51c6",
+    "\u51cc",
+    "\u51dd",
+    "\u51e0",
+    "\u51e1",
+    "\u51e6",
+    "\u51f6",
+    "\u51fa",
+    "\u5200",
+    "\u5203",
+    "\u5206",
+    "\u5207",
+    "\u5208",
+    "\u5211",
+    "\u5215",
+    "\u5217",
+    "\u521d",
+    "\u5224",
+    "\u5225",
+    "\u5229",
+    "\u5230",
+    "\u5236",
+    "\u523a",
+    "\u523b",
+    "\u5243",
+    "\u5247",
+    "\u524a",
+    "\u524d",
+    "\u525b",
+    "\u5263",
+    "\u5265",
+    "\u5270",
+    "\u5272",
+    "\u5287",
+    "\u529b",
+    "\u529f",
+    "\u52a0",
+    "\u52a3",
+    "\u52a9",
+    "\u52aa",
+    "\u52ab",
+    "\u52b1",
+    "\u52b4",
+    "\u52c5",
+    "\u52c7",
+    "\u52d5",
+    "\u52d8",
+    "\u52d9",
+    "\u52dd",
+    "\u52e2",
+    "\u52e4",
+    "\u52e7",
+    "\u52fa",
+    "\u52ff",
+    "\u5301",
+    "\u5302",
+    "\u5305",
+    "\u5315",
+    "\u5316",
+    "\u5317",
+    "\u5320",
+    "\u5323",
+    "\u5339",
+    "\u533b",
+    "\u5341",
+    "\u5343",
+    "\u5345",
+    "\u5347",
+    "\u5348",
+    "\u534a",
+    "\u5351",
+    "\u5352",
+    "\u5353",
+    "\u5357",
+    "\u5358",
+    "\u535a",
+    "\u535c",
+    "\u5360",
+    "\u5366",
+    "\u536f",
+    "\u5370",
+    "\u5371",
+    "\u5373",
+    "\u5374",
+    "\u5375",
+    "\u5378",
+    "\u537f",
+    "\u5384",
+    "\u539a",
+    "\u539f",
+    "\u53ad",
+    "\u53b3",
+    "\u53bb",
+    "\u53c2",
+    "\u53c8",
+    "\u53ca",
+    "\u53cb",
+    "\u53cc",
+    "\u53cd",
+    "\u53ce",
+    "\u53d4",
+    "\u53d6",
+    "\u53d7",
+    "\u53e1",
+    "\u53e2",
+    "\u53e3",
+    "\u53e4",
+    "\u53e5",
+    "\u53ea",
+    "\u53eb",
+    "\u53ec",
+    "\u53ee",
+    "\u53ef",
+    "\u53f0",
+    "\u53f1",
+    "\u53f2",
+    "\u53f3",
+    "\u53f6",
+    "\u53f7",
+    "\u53f8",
+    "\u5404",
+    "\u5408",
+    "\u5409",
+    "\u540a",
+    "\u540c",
+    "\u540d",
+    "\u540e",
+    "\u540f",
+    "\u5410",
+    "\u5411",
+    "\u541b",
+    "\u541d",
+    "\u541f",
+    "\u5426",
+    "\u542b",
+    "\u542d",
+    "\u5438",
+    "\u5439",
+    "\u543b",
+    "\u543c",
+    "\u543e",
+    "\u5442",
+    "\u5446",
+    "\u5449",
+    "\u544a",
+    "\u5451",
+    "\u5468",
+    "\u546a",
+    "\u5473",
+    "\u5475",
+    "\u547c",
+    "\u547d",
+    "\u5480",
+    "\u5484",
+    "\u548c",
+    "\u548e",
+    "\u54b2",
+    "\u54b3",
+    "\u54bd",
+    "\u54c0",
+    "\u54c1",
+    "\u54c9",
+    "\u54e1",
+    "\u54ed",
+    "\u5504",
+    "\u5507",
+    "\u5510",
+    "\u552f",
+    "\u5531",
+    "\u5546",
+    "\u554f",
+    "\u5553",
+    "\u5556",
+    "\u555c",
+    "\u557c",
+    "\u5584",
+    "\u5589",
+    "\u559a",
+    "\u559c",
+    "\u55a7",
+    "\u55aa",
+    "\u55ab",
+    "\u55b0",
+    "\u55b6",
+    "\u55c5",
+    "\u55da",
+    "\u55dc",
+    "\u55df",
+    "\u55e3",
+    "\u5606",
+    "\u5609",
+    "\u5617",
+    "\u5618",
+    "\u5629",
+    "\u5632",
+    "\u5638",
+    "\u5642",
+    "\u564c",
+    "\u5668",
+    "\u566a",
+    "\u567a",
+    "\u56a2",
+    "\u56ae",
+    "\u56db",
+    "\u56de",
+    "\u56e0",
+    "\u56e3",
+    "\u56f0",
+    "\u56f2",
+    "\u56f3",
+    "\u56fa",
+    "\u56fd",
+    "\u5711",
+    "\u5712",
+    "\u571f",
+    "\u5728",
+    "\u5730",
+    "\u5742",
+    "\u574a",
+    "\u5750",
+    "\u5761",
+    "\u576a",
+    "\u5782",
+    "\u57a2",
+    "\u57a3",
+    "\u57cb",
+    "\u57ce",
+    "\u57d2",
+    "\u57f7",
+    "\u57fa",
+    "\u5800",
+    "\u5802",
+    "\u5805",
+    "\u5815",
+    "\u5824",
+    "\u582a",
+    "\u5831",
+    "\u5834",
+    "\u5835",
+    "\u583a",
+    "\u584a",
+    "\u5854",
+    "\u5857",
+    "\u585a",
+    "\u585e",
+    "\u5869",
+    "\u5875",
+    "\u5883",
+    "\u5893",
+    "\u5897",
+    "\u58a8",
+    "\u58c1",
+    "\u58df",
+    "\u58eb",
+    "\u58ee",
+    "\u58f0",
+    "\u58f1",
+    "\u58f2",
+    "\u58f7",
+    "\u58fc",
+    "\u5909",
+    "\u590f",
+    "\u5915",
+    "\u5916",
+    "\u591a",
+    "\u591c",
+    "\u5922",
+    "\u5925",
+    "\u5927",
+    "\u5929",
+    "\u592a",
+    "\u592b",
+    "\u592e",
+    "\u5931",
+    "\u5937",
+    "\u5947",
+    "\u5948",
+    "\u5949",
+    "\u594f",
+    "\u5951",
+    "\u5955",
+    "\u595a",
+    "\u5962",
+    "\u5965",
+    "\u596a",
+    "\u5973",
+    "\u5974",
+    "\u5978",
+    "\u597d",
+    "\u5982",
+    "\u5983",
+    "\u5984",
+    "\u5993",
+    "\u5996",
+    "\u5999",
+    "\u59ac",
+    "\u59b9",
+    "\u59bb",
+    "\u59be",
+    "\u59c9",
+    "\u59cb",
+    "\u59d1",
+    "\u59d3",
+    "\u59d4",
+    "\u59dc",
+    "\u59e8",
+    "\u59eb",
+    "\u59ff",
+    "\u5a01",
+    "\u5a0c",
+    "\u5a11",
+    "\u5a18",
+    "\u5a23",
+    "\u5a35",
+    "\u5a3c",
+    "\u5a46",
+    "\u5a5a",
+    "\u5a66",
+    "\u5a7f",
+    "\u5a92",
+    "\u5a9a",
+    "\u5ac1",
+    "\u5ac9",
+    "\u5acc",
+    "\u5ae1",
+    "\u5ae9",
+    "\u5b09",
+    "\u5b22",
+    "\u5b50",
+    "\u5b54",
+    "\u5b57",
+    "\u5b58",
+    "\u5b5d",
+    "\u5b5f",
+    "\u5b63",
+    "\u5b64",
+    "\u5b66",
+    "\u5b6b",
+    "\u5b85",
+    "\u5b87",
+    "\u5b88",
+    "\u5b89",
+    "\u5b8c",
+    "\u5b97",
+    "\u5b98",
+    "\u5b9a",
+    "\u5b9b",
+    "\u5b9c",
+    "\u5b9d",
+    "\u5b9f",
+    "\u5ba2",
+    "\u5ba3",
+    "\u5ba4",
+    "\u5ba5",
+    "\u5bae",
+    "\u5bb0",
+    "\u5bb3",
+    "\u5bb4",
+    "\u5bb5",
+    "\u5bb6",
+    "\u5bb9",
+    "\u5bbf",
+    "\u5bc2",
+    "\u5bc4",
+    "\u5bc5",
+    "\u5bc6",
+    "\u5bcc",
+    "\u5bd0",
+    "\u5bd2",
+    "\u5bd3",
+    "\u5bd4",
+    "\u5bdb",
+    "\u5bdd",
+    "\u5bdf",
+    "\u5be1",
+    "\u5be7",
+    "\u5be9",
+    "\u5bee",
+    "\u5bf5",
+    "\u5bf8",
+    "\u5bfa",
+    "\u5bfe",
+    "\u5bff",
+    "\u5c01",
+    "\u5c02",
+    "\u5c04",
+    "\u5c06",
+    "\u5c0a",
+    "\u5c0b",
+    "\u5c0e",
+    "\u5c0f",
+    "\u5c11",
+    "\u5c16",
+    "\u5c1a",
+    "\u5c24",
+    "\u5c2d",
+    "\u5c31",
+    "\u5c3a",
+    "\u5c3b",
+    "\u5c3c",
+    "\u5c3d",
+    "\u5c3e",
+    "\u5c3f",
+    "\u5c40",
+    "\u5c45",
+    "\u5c48",
+    "\u5c4a",
+    "\u5c4b",
+    "\u5c4d",
+    "\u5c4e",
+    "\u5c4f",
+    "\u5c51",
+    "\u5c53",
+    "\u5c5e",
+    "\u5c60",
+    "\u5c61",
+    "\u5c65",
+    "\u5c71",
+    "\u5c90",
+    "\u5ca1",
+    "\u5ca9",
+    "\u5cb3",
+    "\u5cb8",
+    "\u5ce8",
+    "\u5cf0",
+    "\u5cf6",
+    "\u5d07",
+    "\u5d0e",
+    "\u5d29",
+    "\u5d50",
+    "\u5d6f",
+    "\u5dba",
+    "\u5dbc",
+    "\u5dcc",
+    "\u5ddd",
+    "\u5dde",
+    "\u5de3",
+    "\u5de5",
+    "\u5de6",
+    "\u5de7",
+    "\u5de8",
+    "\u5deb",
+    "\u5dee",
+    "\u5df1",
+    "\u5df2",
+    "\u5df3",
+    "\u5df4",
+    "\u5dfb",
+    "\u5dfe",
+    "\u5e02",
+    "\u5e03",
+    "\u5e06",
+    "\u5e0c",
+    "\u5e1d",
+    "\u5e2b",
+    "\u5e2d",
+    "\u5e2f",
+    "\u5e30",
+    "\u5e33",
+    "\u5e37",
+    "\u5e38",
+    "\u5e3d",
+    "\u5e45",
+    "\u5e55",
+    "\u5e61",
+    "\u5e63",
+    "\u5e72",
+    "\u5e73",
+    "\u5e74",
+    "\u5e76",
+    "\u5e78",
+    "\u5e7b",
+    "\u5e7c",
+    "\u5e7d",
+    "\u5e7e",
+    "\u5e7f",
+    "\u5e83",
+    "\u5e8a",
+    "\u5e8f",
+    "\u5e95",
+    "\u5e96",
+    "\u5e97",
+    "\u5e9a",
+    "\u5e9c",
+    "\u5ea6",
+    "\u5ea7",
+    "\u5eab",
+    "\u5ead",
+    "\u5eb5",
+    "\u5eb6",
+    "\u5eb7",
+    "\u5ec3",
+    "\u5ec9",
+    "\u5eca",
+    "\u5ed3",
+    "\u5edf",
+    "\u5ef6",
+    "\u5ef7",
+    "\u5efa",
+    "\u5efb",
+    "\u5efc",
+    "\u5eff",
+    "\u5f01",
+    "\u5f0f",
+    "\u5f10",
+    "\u5f13",
+    "\u5f14",
+    "\u5f15",
+    "\u5f18",
+    "\u5f1f",
+    "\u5f25",
+    "\u5f27",
+    "\u5f31",
+    "\u5f35",
+    "\u5f37",
+    "\u5f3e",
+    "\u5f42",
+    "\u5f53",
+    "\u5f62",
+    "\u5f66",
+    "\u5f6b",
+    "\u5f71",
+    "\u5f73",
+    "\u5f79",
+    "\u5f7c",
+    "\u5f80",
+    "\u5f81",
+    "\u5f82",
+    "\u5f84",
+    "\u5f85",
+    "\u5f8b",
+    "\u5f8c",
+    "\u5f92",
+    "\u5f93",
+    "\u5f97",
+    "\u5fa1",
+    "\u5fa9",
+    "\u5fae",
+    "\u5fb3",
+    "\u5fb9",
+    "\u5fbd",
+    "\u5fc3",
+    "\u5fc5",
+    "\u5fcc",
+    "\u5fcd",
+    "\u5fd7",
+    "\u5fd8",
+    "\u5fd9",
+    "\u5fdc",
+    "\u5fdd",
+    "\u5fe0",
+    "\u5feb",
+    "\u5ff0",
+    "\u5ff5",
+    "\u5ffd",
+    "\u6012",
+    "\u6015",
+    "\u6016",
+    "\u601d",
+    "\u6020",
+    "\u6025",
+    "\u6027",
+    "\u6028",
+    "\u602a",
+    "\u6043",
+    "\u604b",
+    "\u6050",
+    "\u6052",
+    "\u6059",
+    "\u605a",
+    "\u605f",
+    "\u6063",
+    "\u6065",
+    "\u6068",
+    "\u6069",
+    "\u606d",
+    "\u606f",
+    "\u6070",
+    "\u6075",
+    "\u6089",
+    "\u608b",
+    "\u6094",
+    "\u609f",
+    "\u60a3",
+    "\u60a6",
+    "\u60a9",
+    "\u60aa",
+    "\u60b2",
+    "\u60b6",
+    "\u60c5",
+    "\u60d1",
+    "\u60da",
+    "\u60dc",
+    "\u60e3",
+    "\u60f3",
+    "\u6101",
+    "\u610f",
+    "\u611a",
+    "\u611b",
+    "\u611f",
+    "\u6127",
+    "\u613d",
+    "\u6148",
+    "\u614b",
+    "\u614c",
+    "\u614e",
+    "\u6155",
+    "\u6162",
+    "\u6165",
+    "\u616e",
+    "\u6170",
+    "\u6173",
+    "\u6176",
+    "\u617e",
+    "\u6182",
+    "\u618e",
+    "\u6190",
+    "\u6191",
+    "\u619a",
+    "\u61a4",
+    "\u61b2",
+    "\u61c7",
+    "\u61d0",
+    "\u61f2",
+    "\u61f8",
+    "\u61fa",
+    "\u620a",
+    "\u620c",
+    "\u620e",
+    "\u6210",
+    "\u6211",
+    "\u6212",
+    "\u6216",
+    "\u6226",
+    "\u622e",
+    "\u622f",
+    "\u6234",
+    "\u6238",
+    "\u623b",
+    "\u623e",
+    "\u623f",
+    "\u6240",
+    "\u6241",
+    "\u6247",
+    "\u6249",
+    "\u624b",
+    "\u624d",
+    "\u6253",
+    "\u6255",
+    "\u6258",
+    "\u625b",
+    "\u6268",
+    "\u626e",
+    "\u6271",
+    "\u6276",
+    "\u627f",
+    "\u6280",
+    "\u6284",
+    "\u628a",
+    "\u6291",
+    "\u6293",
+    "\u6294",
+    "\u6295",
+    "\u6298",
+    "\u629c",
+    "\u629e",
+    "\u62ab",
+    "\u62b1",
+    "\u62b5",
+    "\u62bc",
+    "\u62c5",
+    "\u62cd",
+    "\u62d2",
+    "\u62d9",
+    "\u62db",
+    "\u62dd",
+    "\u62e0",
+    "\u62ec",
+    "\u62ed",
+    "\u62f3",
+    "\u62f5",
+    "\u62f6",
+    "\u62fe",
+    "\u6301",
+    "\u6307",
+    "\u6309",
+    "\u6311",
+    "\u6319",
+    "\u631f",
+    "\u6328",
+    "\u632f",
+    "\u633d",
+    "\u633f",
+    "\u6349",
+    "\u6355",
+    "\u635c",
+    "\u6367",
+    "\u6368",
+    "\u636e",
+    "\u637b",
+    "\u6383",
+    "\u6388",
+    "\u6389",
+    "\u638c",
+    "\u6398",
+    "\u639b",
+    "\u639f",
+    "\u63a1",
+    "\u63a2",
+    "\u63a5",
+    "\u63a8",
+    "\u63b4",
+    "\u63bb",
+    "\u63c3",
+    "\u63c9",
+    "\u63d0",
+    "\u63da",
+    "\u63db",
+    "\u63e1",
+    "\u63fa",
+    "\u640d",
+    "\u6417",
+    "\u643a",
+    "\u6442",
+    "\u6458",
+    "\u6469",
+    "\u647a",
+    "\u6483",
+    "\u64a5",
+    "\u64ab",
+    "\u64ad",
+    "\u64b0",
+    "\u64b2",
+    "\u64b9",
+    "\u64cd",
+    "\u64ec",
+    "\u64f2",
+    "\u652f",
+    "\u6539",
+    "\u653b",
+    "\u653e",
+    "\u653f",
+    "\u6545",
+    "\u6551",
+    "\u6557",
+    "\u6559",
+    "\u6562",
+    "\u6563",
+    "\u6566",
+    "\u656c",
+    "\u6570",
+    "\u6572",
+    "\u6575",
+    "\u6577",
+    "\u6587",
+    "\u6589",
+    "\u658e",
+    "\u6590",
+    "\u6591",
+    "\u6597",
+    "\u6599",
+    "\u659c",
+    "\u65a4",
+    "\u65ac",
+    "\u65ad",
+    "\u65af",
+    "\u65b0",
+    "\u65b9",
+    "\u65bc",
+    "\u65bd",
+    "\u65c5",
+    "\u65cf",
+    "\u65d7",
+    "\u65e2",
+    "\u65e5",
+    "\u65e6",
+    "\u65e7",
+    "\u65e8",
+    "\u65e9",
+    "\u65ec",
+    "\u6606",
+    "\u6607",
+    "\u660c",
+    "\u660e",
+    "\u660f",
+    "\u6613",
+    "\u6614",
+    "\u661f",
+    "\u6625",
+    "\u6628",
+    "\u662f",
+    "\u663c",
+    "\u6642",
+    "\u664b",
+    "\u6652",
+    "\u6666",
+    "\u6669",
+    "\u666e",
+    "\u666f",
+    "\u6674",
+    "\u667a",
+    "\u6681",
+    "\u6687",
+    "\u6691",
+    "\u6696",
+    "\u6697",
+    "\u66a6",
+    "\u66ab",
+    "\u66ae",
+    "\u66b4",
+    "\u66c7",
+    "\u66d9",
+    "\u66da",
+    "\u66dc",
+    "\u66dd",
+    "\u66f0",
+    "\u66f2",
+    "\u66f3",
+    "\u66f4",
+    "\u66f8",
+    "\u66f9",
+    "\u66fd",
+    "\u66ff",
+    "\u6700",
+    "\u6708",
+    "\u6709",
+    "\u670b",
+    "\u670d",
+    "\u6714",
+    "\u6715",
+    "\u671b",
+    "\u671d",
+    "\u671f",
+    "\u6727",
+    "\u6728",
+    "\u672a",
+    "\u672b",
+    "\u672c",
+    "\u672d",
+    "\u6731",
+    "\u6734",
+    "\u673d",
+    "\u6749",
+    "\u674e",
+    "\u674f",
+    "\u6750",
+    "\u6751",
+    "\u6753",
+    "\u6756",
+    "\u675c",
+    "\u675f",
+    "\u6761",
+    "\u6763",
+    "\u6765",
+    "\u676f",
+    "\u6771",
+    "\u6775",
+    "\u6777",
+    "\u677e",
+    "\u677f",
+    "\u6795",
+    "\u6797",
+    "\u679a",
+    "\u679c",
+    "\u679d",
+    "\u67af",
+    "\u67b6",
+    "\u67b8",
+    "\u67c4",
+    "\u67ca",
+    "\u67cf",
+    "\u67d0",
+    "\u67d1",
+    "\u67d3",
+    "\u67d4",
+    "\u67d8",
+    "\u67da",
+    "\u67f1",
+    "\u67f3",
+    "\u67f4",
+    "\u67f5",
+    "\u67ff",
+    "\u6804",
+    "\u6817",
+    "\u682a",
+    "\u6839",
+    "\u683c",
+    "\u683d",
+    "\u6841",
+    "\u6842",
+    "\u6843",
+    "\u6848",
+    "\u6850",
+    "\u6851",
+    "\u6854",
+    "\u685c",
+    "\u685d",
+    "\u685f",
+    "\u6867",
+    "\u6876",
+    "\u6881",
+    "\u6885",
+    "\u6893",
+    "\u6894",
+    "\u6897",
+    "\u689f",
+    "\u68a2",
+    "\u68a8",
+    "\u68b6",
+    "\u68c4",
+    "\u68ca",
+    "\u68d2",
+    "\u68d7",
+    "\u68d8",
+    "\u68da",
+    "\u68df",
+    "\u68ee",
+    "\u68f2",
+    "\u68f9",
+    "\u68fa",
+    "\u6900",
+    "\u690d",
+    "\u690e",
+    "\u6912",
+    "\u691c",
+    "\u6939",
+    "\u693d",
+    "\u693f",
+    "\u694a",
+    "\u695a",
+    "\u6960",
+    "\u696d",
+    "\u6975",
+    "\u697c",
+    "\u697d",
+    "\u6982",
+    "\u698e",
+    "\u699b",
+    "\u69b4",
+    "\u69c3",
+    "\u69cb",
+    "\u69cc",
+    "\u69d8",
+    "\u69d9",
+    "\u6a0b",
+    "\u6a19",
+    "\u6a21",
+    "\u6a29",
+    "\u6a2a",
+    "\u6a39",
+    "\u6a3d",
+    "\u6a4b",
+    "\u6a58",
+    "\u6a59",
+    "\u6a5f",
+    "\u6a80",
+    "\u6ac3",
+    "\u6adb",
+    "\u6b1d",
+    "\u6b20",
+    "\u6b21",
+    "\u6b32",
+    "\u6b3a",
+    "\u6b4c",
+    "\u6b4e",
+    "\u6b53",
+    "\u6b5f",
+    "\u6b62",
+    "\u6b63",
+    "\u6b64",
+    "\u6b66",
+    "\u6b69",
+    "\u6b6f",
+    "\u6b73",
+    "\u6b74",
+    "\u6b7b",
+    "\u6b86",
+    "\u6b8a",
+    "\u6b8b",
+    "\u6b96",
+    "\u6bb5",
+    "\u6bba",
+    "\u6bbb",
+    "\u6bbf",
+    "\u6bcb",
+    "\u6bcd",
+    "\u6bce",
+    "\u6bd2",
+    "\u6bd4",
+    "\u6bd8",
+    "\u6bdb",
+    "\u6beb",
+    "\u6bec",
+    "\u6c0f",
+    "\u6c11",
+    "\u6c17",
+    "\u6c34",
+    "\u6c37",
+    "\u6c38",
+    "\u6c40",
+    "\u6c41",
+    "\u6c42",
+    "\u6c50",
+    "\u6c57",
+    "\u6c5a",
+    "\u6c5d",
+    "\u6c5f",
+    "\u6c60",
+    "\u6c70",
+    "\u6c72",
+    "\u6c7a",
+    "\u6c88",
+    "\u6c93",
+    "\u6c96",
+    "\u6c99",
+    "\u6ca1",
+    "\u6ca2",
+    "\u6cab",
+    "\u6cb3",
+    "\u6cb8",
+    "\u6cb9",
+    "\u6cbb",
+    "\u6cbc",
+    "\u6cbd",
+    "\u6cc1",
+    "\u6cc9",
+    "\u6cca",
+    "\u6cd5",
+    "\u6ce1",
+    "\u6ce2",
+    "\u6ce3",
+    "\u6ce5",
+    "\u6ce8",
+    "\u6cf0",
+    "\u6d0b",
+    "\u6d12",
+    "\u6d17",
+    "\u6d1b",
+    "\u6d1e",
+    "\u6d25",
+    "\u6d29",
+    "\u6d2a",
+    "\u6d32",
+    "\u6d3b",
+    "\u6d3e",
+    "\u6d41",
+    "\u6d44",
+    "\u6d45",
+    "\u6d5c",
+    "\u6d66",
+    "\u6d69",
+    "\u6d6a",
+    "\u6d6e",
+    "\u6d74",
+    "\u6d77",
+    "\u6d78",
+    "\u6d88",
+    "\u6d95",
+    "\u6d99",
+    "\u6daf",
+    "\u6db2",
+    "\u6dbc",
+    "\u6dc0",
+    "\u6dcb",
+    "\u6de1",
+    "\u6deb",
+    "\u6df1",
+    "\u6df5",
+    "\u6df7",
+    "\u6dfb",
+    "\u6e05",
+    "\u6e07",
+    "\u6e08",
+    "\u6e0b",
+    "\u6e13",
+    "\u6e1b",
+    "\u6e20",
+    "\u6e21",
+    "\u6e29",
+    "\u6e2c",
+    "\u6e38",
+    "\u6e3e",
+    "\u6e4a",
+    "\u6e56",
+    "\u6e5b",
+    "\u6e67",
+    "\u6e6f",
+    "\u6e7f",
+    "\u6e80",
+    "\u6e8f",
+    "\u6e90",
+    "\u6e9c",
+    "\u6e9d",
+    "\u6ec5",
+    "\u6ed1",
+    "\u6edd",
+    "\u6f01",
+    "\u6f02",
+    "\u6f06",
+    "\u6f09",
+    "\u6f0f",
+    "\u6f14",
+    "\u6f15",
+    "\u6f22",
+    "\u6f2b",
+    "\u6f2c",
+    "\u6f38",
+    "\u6f45",
+    "\u6f54",
+    "\u6f5c",
+    "\u6f64",
+    "\u6f6e",
+    "\u6f70",
+    "\u6fc0",
+    "\u6fc1",
+    "\u6fc3",
+    "\u6fe1",
+    "\u6fe4",
+    "\u6fef",
+    "\u702c",
+    "\u706b",
+    "\u706f",
+    "\u7070",
+    "\u7078",
+    "\u707d",
+    "\u7089",
+    "\u708a",
+    "\u708e",
+    "\u7092",
+    "\u7099",
+    "\u70ad",
+    "\u70ae",
+    "\u70b9",
+    "\u70ba",
+    "\u70c8",
+    "\u70cf",
+    "\u7109",
+    "\u711a",
+    "\u7121",
+    "\u7126",
+    "\u7136",
+    "\u713c",
+    "\u7149",
+    "\u714e",
+    "\u7159",
+    "\u7164",
+    "\u7167",
+    "\u7169",
+    "\u716e",
+    "\u718a",
+    "\u719f",
+    "\u71a8",
+    "\u71ac",
+    "\u71b1",
+    "\u71c3",
+    "\u71d5",
+    "\u71d7",
+    "\u71e0",
+    "\u71e5",
+    "\u71ed",
+    "\u71f5",
+    "\u722a",
+    "\u7230",
+    "\u7236",
+    "\u723a",
+    "\u723e",
+    "\u7247",
+    "\u7248",
+    "\u724c",
+    "\u7259",
+    "\u725b",
+    "\u7261",
+    "\u7262",
+    "\u7267",
+    "\u7269",
+    "\u727d",
+    "\u72a2",
+    "\u72ac",
+    "\u72af",
+    "\u72b6",
+    "\u72c2",
+    "\u72ce",
+    "\u72d0",
+    "\u72d7",
+    "\u72e9",
+    "\u72ec",
+    "\u72ed",
+    "\u72f8",
+    "\u72fc",
+    "\u731b",
+    "\u731c",
+    "\u731f",
+    "\u7325",
+    "\u7329",
+    "\u732a",
+    "\u732b",
+    "\u732e",
+    "\u7334",
+    "\u7336",
+    "\u733f",
+    "\u7344",
+    "\u7345",
+    "\u7363",
+    "\u7372",
+    "\u7384",
+    "\u7387",
+    "\u7389",
+    "\u738b",
+    "\u73cd",
+    "\u73e0",
+    "\u73fe",
+    "\u7403",
+    "\u7406",
+    "\u7409",
+    "\u7434",
+    "\u745e",
+    "\u7460",
+    "\u7483",
+    "\u74bd",
+    "\u74dc",
+    "\u74e2",
+    "\u74e6",
+    "\u74f6",
+    "\u7518",
+    "\u751a",
+    "\u751f",
+    "\u7523",
+    "\u7528",
+    "\u752b",
+    "\u7530",
+    "\u7531",
+    "\u7532",
+    "\u7533",
+    "\u7537",
+    "\u753a",
+    "\u753b",
+    "\u754c",
+    "\u754f",
+    "\u7551",
+    "\u7559",
+    "\u755c",
+    "\u755d",
+    "\u7560",
+    "\u7562",
+    "\u7565",
+    "\u756a",
+    "\u7570",
+    "\u7573",
+    "\u7577",
+    "\u757f",
+    "\u758b",
+    "\u758e",
+    "\u7591",
+    "\u75ab",
+    "\u75b1",
+    "\u75b2",
+    "\u75b3",
+    "\u75b5",
+    "\u75be",
+    "\u75c5",
+    "\u75c7",
+    "\u75cd",
+    "\u75db",
+    "\u75e9",
+    "\u75f4",
+    "\u7621",
+    "\u7624",
+    "\u7642",
+    "\u7656",
+    "\u766a",
+    "\u7678",
+    "\u767a",
+    "\u767b",
+    "\u767d",
+    "\u767e",
+    "\u7684",
+    "\u7686",
+    "\u7687",
+    "\u76ae",
+    "\u76bf",
+    "\u76c6",
+    "\u76ca",
+    "\u76d7",
+    "\u76db",
+    "\u76de",
+    "\u76df",
+    "\u76e4",
+    "\u76e5",
+    "\u76e7",
+    "\u76ee",
+    "\u76f2",
+    "\u76f4",
+    "\u76f8",
+    "\u76fe",
+    "\u7701",
+    "\u7709",
+    "\u770b",
+    "\u770c",
+    "\u771f",
+    "\u7720",
+    "\u773c",
+    "\u7740",
+    "\u7761",
+    "\u7763",
+    "\u7766",
+    "\u7768",
+    "\u77bb",
+    "\u77e2",
+    "\u77e3",
+    "\u77e5",
+    "\u77ed",
+    "\u77f3",
+    "\u7802",
+    "\u780d",
+    "\u7814",
+    "\u7815",
+    "\u7825",
+    "\u7834",
+    "\u786b",
+    "\u786f",
+    "\u7881",
+    "\u7893",
+    "\u7897",
+    "\u78a7",
+    "\u78c1",
+    "\u78e8",
+    "\u78ef",
+    "\u7912",
+    "\u792b",
+    "\u793a",
+    "\u793c",
+    "\u793e",
+    "\u7947",
+    "\u7948",
+    "\u7956",
+    "\u795d",
+    "\u795e",
+    "\u795f",
+    "\u7960",
+    "\u7962",
+    "\u7965",
+    "\u796d",
+    "\u7977",
+    "\u7980",
+    "\u7981",
+    "\u7984",
+    "\u7985",
+    "\u798a",
+    "\u798d",
+    "\u798f",
+    "\u79a6",
+    "\u79bd",
+    "\u79bf",
+    "\u79c0",
+    "\u79c1",
+    "\u79cb",
+    "\u79d1",
+    "\u79d8",
+    "\u79e1",
+    "\u79e4",
+    "\u79e6",
+    "\u79f0",
+    "\u79fb",
+    "\u7a00",
+    "\u7a0b",
+    "\u7a17",
+    "\u7a1a",
+    "\u7a2e",
+    "\u7a32",
+    "\u7a3d",
+    "\u7a3f",
+    "\u7a40",
+    "\u7a42",
+    "\u7a4d",
+    "\u7a62",
+    "\u7a74",
+    "\u7a76",
+    "\u7a7a",
+    "\u7a7f",
+    "\u7a81",
+    "\u7a93",
+    "\u7a9f",
+    "\u7aae",
+    "\u7ab6",
+    "\u7aba",
+    "\u7ac3",
+    "\u7ac4",
+    "\u7aca",
+    "\u7acb",
+    "\u7adc",
+    "\u7adf",
+    "\u7ae0",
+    "\u7ae5",
+    "\u7aed",
+    "\u7aef",
+    "\u7af6",
+    "\u7af9",
+    "\u7afa",
+    "\u7aff",
+    "\u7b04",
+    "\u7b0a",
+    "\u7b11",
+    "\u7b18",
+    "\u7b1b",
+    "\u7b20",
+    "\u7b26",
+    "\u7b2c",
+    "\u7b39",
+    "\u7b46",
+    "\u7b48",
+    "\u7b49",
+    "\u7b4b",
+    "\u7b4d",
+    "\u7b4e",
+    "\u7b51",
+    "\u7b52",
+    "\u7b54",
+    "\u7b56",
+    "\u7b65",
+    "\u7b87",
+    "\u7b92",
+    "\u7b94",
+    "\u7b95",
+    "\u7b97",
+    "\u7ba1",
+    "\u7baa",
+    "\u7bad",
+    "\u7bb1",
+    "\u7bb8",
+    "\u7bc0",
+    "\u7bc4",
+    "\u7bc7",
+    "\u7bc9",
+    "\u7be0",
+    "\u7be4",
+    "\u7be9",
+    "\u7bed",
+    "\u7c00",
+    "\u7c11",
+    "\u7c21",
+    "\u7c3e",
+    "\u7c4d",
+    "\u7c73",
+    "\u7c7e",
+    "\u7c83",
+    "\u7c89",
+    "\u7c8b",
+    "\u7c92",
+    "\u7c95",
+    "\u7c97",
+    "\u7c9f",
+    "\u7ca5",
+    "\u7ca7",
+    "\u7cb3",
+    "\u7cbd",
+    "\u7cbe",
+    "\u7cc0",
+    "\u7cca",
+    "\u7cd5",
+    "\u7cd6",
+    "\u7cde",
+    "\u7cdf",
+    "\u7ce0",
+    "\u7ce7",
+    "\u7cef",
+    "\u7cf8",
+    "\u7cfa",
+    "\u7cfb",
+    "\u7d00",
+    "\u7d04",
+    "\u7d05",
+    "\u7d0b",
+    "\u7d0d",
+    "\u7d10",
+    "\u7d17",
+    "\u7d19",
+    "\u7d1b",
+    "\u7d20",
+    "\u7d22",
+    "\u7d2b",
+    "\u7d2c",
+    "\u7d2f",
+    "\u7d30",
+    "\u7d39",
+    "\u7d3a",
+    "\u7d42",
+    "\u7d43",
+    "\u7d44",
+    "\u7d4c",
+    "\u7d50",
+    "\u7d5e",
+    "\u7d66",
+    "\u7d71",
+    "\u7d75",
+    "\u7d76",
+    "\u7d79",
+    "\u7d99",
+    "\u7d9a",
+    "\u7db1",
+    "\u7db2",
+    "\u7db4",
+    "\u7db8",
+    "\u7dba",
+    "\u7dbe",
+    "\u7dbf",
+    "\u7dcb",
+    "\u7dcf",
+    "\u7dd1",
+    "\u7dd2",
+    "\u7dda",
+    "\u7de8",
+    "\u7de9",
+    "\u7dec",
+    "\u7df4",
+    "\u7e01",
+    "\u7e04",
+    "\u7e1b",
+    "\u7e21",
+    "\u7e23",
+    "\u7e26",
+    "\u7e2b",
+    "\u7e2e",
+    "\u7e41",
+    "\u7e45",
+    "\u7e4b",
+    "\u7e54",
+    "\u7e6d",
+    "\u7e70",
+    "\u7e7b",
+    "\u7e8f",
+    "\u7e94",
+    "\u7e9c",
+    "\u7f6a",
+    "\u7f6e",
+    "\u7f70",
+    "\u7f75",
+    "\u7f77",
+    "\u7f85",
+    "\u7f8a",
+    "\u7f8e",
+    "\u7fa4",
+    "\u7fa8",
+    "\u7fa9",
+    "\u7fb9",
+    "\u7fbd",
+    "\u7fc1",
+    "\u7fc5",
+    "\u7fcc",
+    "\u7fd2",
+    "\u7fd4",
+    "\u7fe0",
+    "\u7feb",
+    "\u7ffa",
+    "\u7ffb",
+    "\u7ffc",
+    "\u8001",
+    "\u8003",
+    "\u8005",
+    "\u800c",
+    "\u8012",
+    "\u8015",
+    "\u8017",
+    "\u8033",
+    "\u804a",
+    "\u8056",
+    "\u805a",
+    "\u805e",
+    "\u8073",
+    "\u8074",
+    "\u8077",
+    "\u8086",
+    "\u8089",
+    "\u808c",
+    "\u8096",
+    "\u809a",
+    "\u809d",
+    "\u80a1",
+    "\u80a5",
+    "\u80a9",
+    "\u80af",
+    "\u80b2",
+    "\u80b4",
+    "\u80c3",
+    "\u80c6",
+    "\u80cc",
+    "\u80ce",
+    "\u80de",
+    "\u80e1",
+    "\u80f4",
+    "\u80f8",
+    "\u80fd",
+    "\u8102",
+    "\u8107",
+    "\u8108",
+    "\u810a",
+    "\u811a",
+    "\u8129",
+    "\u8131",
+    "\u8133",
+    "\u813e",
+    "\u814e",
+    "\u8150",
+    "\u8155",
+    "\u8165",
+    "\u816b",
+    "\u816e",
+    "\u8170",
+    "\u8178",
+    "\u8179",
+    "\u817f",
+    "\u8188",
+    "\u818f",
+    "\u819a",
+    "\u819d",
+    "\u81b3",
+    "\u81be",
+    "\u81cd",
+    "\u81d8",
+    "\u81e3",
+    "\u81e5",
+    "\u81e8",
+    "\u81ea",
+    "\u81ed",
+    "\u81f3",
+    "\u81f4",
+    "\u81fc",
+    "\u8201",
+    "\u8202",
+    "\u8208",
+    "\u820c",
+    "\u820e",
+    "\u8217",
+    "\u821b",
+    "\u821c",
+    "\u821e",
+    "\u821f",
+    "\u822c",
+    "\u8239",
+    "\u8258",
+    "\u826f",
+    "\u8271",
+    "\u8272",
+    "\u8276",
+    "\u8278",
+    "\u828b",
+    "\u8292",
+    "\u829d",
+    "\u82a5",
+    "\u82a6",
+    "\u82ad",
+    "\u82b1",
+    "\u82b3",
+    "\u82b8",
+    "\u82b9",
+    "\u82bd",
+    "\u82d4",
+    "\u82d7",
+    "\u82de",
+    "\u82df",
+    "\u82e5",
+    "\u82e6",
+    "\u82f1",
+    "\u8302",
+    "\u8304",
+    "\u8305",
+    "\u830e",
+    "\u8317",
+    "\u8328",
+    "\u8332",
+    "\u8336",
+    "\u8338",
+    "\u8339",
+    "\u8349",
+    "\u834a",
+    "\u8352",
+    "\u8358",
+    "\u8377",
+    "\u837b",
+    "\u839a",
+    "\u839e",
+    "\u83a8",
+    "\u83ab",
+    "\u83b1",
+    "\u83c5",
+    "\u83ca",
+    "\u83d3",
+    "\u83d6",
+    "\u83dc",
+    "\u83df",
+    "\u83e9",
+    "\u83ef",
+    "\u83f0",
+    "\u83f1",
+    "\u8429",
+    "\u8431",
+    "\u843d",
+    "\u8449",
+    "\u8457",
+    "\u845b",
+    "\u8463",
+    "\u8466",
+    "\u846c",
+    "\u846d",
+    "\u8471",
+    "\u8475",
+    "\u847a",
+    "\u8490",
+    "\u8494",
+    "\u8499",
+    "\u849c",
+    "\u84b2",
+    "\u84b8",
+    "\u84bf",
+    "\u84cb",
+    "\u84ec",
+    "\u84ee",
+    "\u84fc",
+    "\u8513",
+    "\u8521",
+    "\u8526",
+    "\u852d",
+    "\u8535",
+    "\u8549",
+    "\u854e",
+    "\u8557",
+    "\u8568",
+    "\u8569",
+    "\u856a",
+    "\u8577",
+    "\u8584",
+    "\u85a6",
+    "\u85a9",
+    "\u85aa",
+    "\u85ab",
+    "\u85ac",
+    "\u85ae",
+    "\u85af",
+    "\u85ba",
+    "\u85c1",
+    "\u85e4",
+    "\u85fb",
+    "\u8607",
+    "\u862d",
+    "\u863f",
+    "\u864e",
+    "\u865a",
+    "\u865e",
+    "\u866b",
+    "\u866c",
+    "\u8671",
+    "\u868a",
+    "\u8695",
+    "\u86a4",
+    "\u86ab",
+    "\u86c7",
+    "\u86c9",
+    "\u86cd",
+    "\u86ce",
+    "\u86d9",
+    "\u86db",
+    "\u86e4",
+    "\u86ee",
+    "\u86f8",
+    "\u86fe",
+    "\u8700",
+    "\u8702",
+    "\u8706",
+    "\u8711",
+    "\u8718",
+    "\u871c",
+    "\u8737",
+    "\u873b",
+    "\u8749",
+    "\u874b",
+    "\u8766",
+    "\u8776",
+    "\u877f",
+    "\u87ba",
+    "\u87c4",
+    "\u87c7",
+    "\u87e0",
+    "\u87f9",
+    "\u87fb",
+    "\u880e",
+    "\u8840",
+    "\u8846",
+    "\u884c",
+    "\u8852",
+    "\u8853",
+    "\u8857",
+    "\u885b",
+    "\u885d",
+    "\u8863",
+    "\u8868",
+    "\u8870",
+    "\u887e",
+    "\u887f",
+    "\u8882",
+    "\u8888",
+    "\u888b",
+    "\u888d",
+    "\u8896",
+    "\u88ab",
+    "\u88b4",
+    "\u88b7",
+    "\u88c1",
+    "\u88c2",
+    "\u88c5",
+    "\u88cf",
+    "\u88dc",
+    "\u88df",
+    "\u88f3",
+    "\u88f8",
+    "\u88fd",
+    "\u88fe",
+    "\u890c",
+    "\u8912",
+    "\u895f",
+    "\u897f",
+    "\u8981",
+    "\u8986",
+    "\u898b",
+    "\u898f",
+    "\u8996",
+    "\u8997",
+    "\u899a",
+    "\u89a7",
+    "\u89aa",
+    "\u89b3",
+    "\u89d2",
+    "\u89dc",
+    "\u89e3",
+    "\u89e6",
+    "\u8a00",
+    "\u8a08",
+    "\u8a0e",
+    "\u8a13",
+    "\u8a17",
+    "\u8a18",
+    "\u8a1b",
+    "\u8a1f",
+    "\u8a2a",
+    "\u8a2d",
+    "\u8a31",
+    "\u8a33",
+    "\u8a34",
+    "\u8a3b",
+    "\u8a3c",
+    "\u8a54",
+    "\u8a55",
+    "\u8a5e",
+    "\u8a60",
+    "\u8a63",
+    "\u8a66",
+    "\u8a69",
+    "\u8a6b",
+    "\u8a6e",
+    "\u8a70",
+    "\u8a71",
+    "\u8a73",
+    "\u8a75",
+    "\u8a82",
+    "\u8a87",
+    "\u8a89",
+    "\u8a8c",
+    "\u8a8d",
+    "\u8a93",
+    "\u8a98",
+    "\u8a9e",
+    "\u8aa0",
+    "\u8aa4",
+    "\u8aac",
+    "\u8aad",
+    "\u8ab0",
+    "\u8ab2",
+    "\u8ab9",
+    "\u8abf",
+    "\u8ac7",
+    "\u8acb",
+    "\u8acc",
+    "\u8ad6",
+    "\u8ada",
+    "\u8ae7",
+    "\u8aed",
+    "\u8af1",
+    "\u8af7",
+    "\u8af8",
+    "\u8afa",
+    "\u8b00",
+    "\u8b02",
+    "\u8b0e",
+    "\u8b17",
+    "\u8b19",
+    "\u8b1b",
+    "\u8b1d",
+    "\u8b21",
+    "\u8b58",
+    "\u8b5a",
+    "\u8b6c",
+    "\u8b70",
+    "\u8b72",
+    "\u8b77",
+    "\u8b83",
+    "\u8b90",
+    "\u8b92",
+    "\u8c37",
+    "\u8c46",
+    "\u8c48",
+    "\u8c4a",
+    "\u8c61",
+    "\u8c8c",
+    "\u8c9d",
+    "\u8c9e",
+    "\u8ca0",
+    "\u8ca1",
+    "\u8ca2",
+    "\u8ca7",
+    "\u8ca8",
+    "\u8caa",
+    "\u8cab",
+    "\u8cac",
+    "\u8caf",
+    "\u8cb0",
+    "\u8cb4",
+    "\u8cb7",
+    "\u8cb8",
+    "\u8cbb",
+    "\u8cbc",
+    "\u8cc0",
+    "\u8cc3",
+    "\u8cc4",
+    "\u8cc7",
+    "\u8cca",
+    "\u8cce",
+    "\u8cd1",
+    "\u8cd3",
+    "\u8cdc",
+    "\u8cde",
+    "\u8ce2",
+    "\u8cea",
+    "\u8d08",
+    "\u8d14",
+    "\u8d16",
+    "\u8d64",
+    "\u8d66",
+    "\u8d70",
+    "\u8d74",
+    "\u8d77",
+    "\u8d8a",
+    "\u8d99",
+    "\u8da3",
+    "\u8db3",
+    "\u8de1",
+    "\u8de8",
+    "\u8dea",
+    "\u8def",
+    "\u8df3",
+    "\u8e0a",
+    "\u8e0f",
+    "\u8e30",
+    "\u8e44",
+    "\u8e8d",
+    "\u8eab",
+    "\u8eaf",
+    "\u8eca",
+    "\u8ecd",
+    "\u8ed2",
+    "\u8ee2",
+    "\u8ef8",
+    "\u8efd",
+    "\u8f09",
+    "\u8f14",
+    "\u8f19",
+    "\u8f1d",
+    "\u8f29",
+    "\u8f2a",
+    "\u8f2f",
+    "\u8f3f",
+    "\u8f9b",
+    "\u8f9e",
+    "\u8fb0",
+    "\u8fb1",
+    "\u8fb2",
+    "\u8fba",
+    "\u8fbb",
+    "\u8fbc",
+    "\u8fc4",
+    "\u8fce",
+    "\u8fd1",
+    "\u8fd4",
+    "\u8fda",
+    "\u8fe6",
+    "\u8feb",
+    "\u8ff0",
+    "\u8ff7",
+    "\u8ffd",
+    "\u9000",
+    "\u9001",
+    "\u9003",
+    "\u9006",
+    "\u900f",
+    "\u9010",
+    "\u9014",
+    "\u9017",
+    "\u9019",
+    "\u901a",
+    "\u901e",
+    "\u901f",
+    "\u9020",
+    "\u9022",
+    "\u9023",
+    "\u9032",
+    "\u9038",
+    "\u9041",
+    "\u9042",
+    "\u9045",
+    "\u9047",
+    "\u904a",
+    "\u904b",
+    "\u904d",
+    "\u904e",
+    "\u9053",
+    "\u9054",
+    "\u9055",
+    "\u9060",
+    "\u9063",
+    "\u9065",
+    "\u9069",
+    "\u9078",
+    "\u907a",
+    "\u907d",
+    "\u907f",
+    "\u9084",
+    "\u9091",
+    "\u90a3",
+    "\u90aa",
+    "\u90ce",
+    "\u90e1",
+    "\u90e8",
+    "\u90ed",
+    "\u90f7",
+    "\u90fd",
+    "\u9119",
+    "\u914c",
+    "\u914d",
+    "\u9152",
+    "\u9154",
+    "\u9162",
+    "\u9177",
+    "\u9192",
+    "\u919c",
+    "\u91a4",
+    "\u91c7",
+    "\u91c8",
+    "\u91cc",
+    "\u91cd",
+    "\u91ce",
+    "\u91cf",
+    "\u91d1",
+    "\u91d8",
+    "\u91dc",
+    "\u91dd",
+    "\u91e3",
+    "\u920e",
+    "\u9234",
+    "\u9244",
+    "\u9262",
+    "\u9266",
+    "\u927e",
+    "\u9280",
+    "\u9285",
+    "\u9298",
+    "\u929a",
+    "\u92ad",
+    "\u92e4",
+    "\u92f8",
+    "\u9320",
+    "\u9326",
+    "\u932b",
+    "\u932c",
+    "\u9332",
+    "\u934b",
+    "\u935b",
+    "\u936c",
+    "\u938c",
+    "\u93a7",
+    "\u93ae",
+    "\u93e1",
+    "\u9418",
+    "\u9419",
+    "\u9451",
+    "\u9453",
+    "\u9475",
+    "\u947f",
+    "\u9577",
+    "\u9580",
+    "\u9583",
+    "\u9589",
+    "\u958b",
+    "\u958f",
+    "\u9591",
+    "\u9593",
+    "\u9594",
+    "\u9599",
+    "\u95a2",
+    "\u95a8",
+    "\u95c7",
+    "\u95d5",
+    "\u95e2",
+    "\u9632",
+    "\u963f",
+    "\u9640",
+    "\u9644",
+    "\u964d",
+    "\u9650",
+    "\u9662",
+    "\u9663",
+    "\u9664",
+    "\u9665",
+    "\u966a",
+    "\u9670",
+    "\u9673",
+    "\u9675",
+    "\u9676",
+    "\u9678",
+    "\u967d",
+    "\u9685",
+    "\u9688",
+    "\u968e",
+    "\u968f",
+    "\u9694",
+    "\u9699",
+    "\u969b",
+    "\u969c",
+    "\u96a0",
+    "\u96a3",
+    "\u96b7",
+    "\u96ba",
+    "\u96bb",
+    "\u96bc",
+    "\u96c0",
+    "\u96c1",
+    "\u96c4",
+    "\u96c5",
+    "\u96c6",
+    "\u96c9",
+    "\u96cc",
+    "\u96d1",
+    "\u96d6",
+    "\u96db",
+    "\u96e2",
+    "\u96e3",
+    "\u96e8",
+    "\u96ea",
+    "\u96eb",
+    "\u96f2",
+    "\u96f6",
+    "\u96f7",
+    "\u96f9",
+    "\u96fb",
+    "\u9700",
+    "\u9707",
+    "\u970a",
+    "\u970e",
+    "\u9716",
+    "\u971c",
+    "\u971e",
+    "\u9727",
+    "\u9730",
+    "\u9732",
+    "\u9752",
+    "\u9759",
+    "\u975e",
+    "\u9761",
+    "\u9762",
+    "\u9769",
+    "\u976d",
+    "\u9781",
+    "\u9786",
+    "\u978b",
+    "\u978d",
+    "\u9798",
+    "\u97a0",
+    "\u97ad",
+    "\u97d3",
+    "\u97f3",
+    "\u97ff",
+    "\u9802",
+    "\u9803",
+    "\u9806",
+    "\u9808",
+    "\u9810",
+    "\u9813",
+    "\u9818",
+    "\u982c",
+    "\u982d",
+    "\u9838",
+    "\u983b",
+    "\u983c",
+    "\u9846",
+    "\u984c",
+    "\u984d",
+    "\u9854",
+    "\u9855",
+    "\u9858",
+    "\u985b",
+    "\u985e",
+    "\u9867",
+    "\u98a8",
+    "\u98db",
+    "\u98df",
+    "\u98e2",
+    "\u98e9",
+    "\u98ef",
+    "\u98f2",
+    "\u98f4",
+    "\u98fc",
+    "\u98fd",
+    "\u98fe",
+    "\u9905",
+    "\u990a",
+    "\u990c",
+    "\u9913",
+    "\u9921",
+    "\u9928",
+    "\u9940",
+    "\u9945",
+    "\u9957",
+    "\u9996",
+    "\u9999",
+    "\u99ac",
+    "\u99b3",
+    "\u99b4",
+    "\u99c4",
+    "\u99c5",
+    "\u99c6",
+    "\u99d2",
+    "\u99d5",
+    "\u99ed",
+    "\u99ff",
+    "\u9a0e",
+    "\u9a12",
+    "\u9a13",
+    "\u9a19",
+    "\u9a28",
+    "\u9a55",
+    "\u9a5a",
+    "\u9a6a",
+    "\u9aa8",
+    "\u9ab8",
+    "\u9ad8",
+    "\u9aea",
+    "\u9aed",
+    "\u9af7",
+    "\u9b1f",
+    "\u9b22",
+    "\u9b3c",
+    "\u9b41",
+    "\u9b42",
+    "\u9b45",
+    "\u9b4f",
+    "\u9b54",
+    "\u9b5a",
+    "\u9b6f",
+    "\u9b8e",
+    "\u9b91",
+    "\u9b92",
+    "\u9b93",
+    "\u9ba0",
+    "\u9ba8",
+    "\u9bab",
+    "\u9bad",
+    "\u9bae",
+    "\u9bb9",
+    "\u9bc9",
+    "\u9bd6",
+    "\u9bdb",
+    "\u9be8",
+    "\u9bf2",
+    "\u9bf7",
+    "\u9c06",
+    "\u9c0c",
+    "\u9c24",
+    "\u9c2d",
+    "\u9c2f",
+    "\u9c39",
+    "\u9c3b",
+    "\u9c48",
+    "\u9c52",
+    "\u9c57",
+    "\u9c60",
+    "\u9c63",
+    "\u9c67",
+    "\u9c78",
+    "\u9ce5",
+    "\u9ce9",
+    "\u9cf0",
+    "\u9cf3",
+    "\u9cf4",
+    "\u9cf6",
+    "\u9d28",
+    "\u9d2b",
+    "\u9d2c",
+    "\u9d3b",
+    "\u9d46",
+    "\u9d59",
+    "\u9d5c",
+    "\u9d5d",
+    "\u9d89",
+    "\u9d8f",
+    "\u9db4",
+    "\u9df2",
+    "\u9df9",
+    "\u9dfa",
+    "\u9e7f",
+    "\u9e81",
+    "\u9e93",
+    "\u9e97",
+    "\u9ea6",
+    "\u9ea9",
+    "\u9eb9",
+    "\u9eba",
+    "\u9ebb",
+    "\u9ec4",
+    "\u9ecd",
+    "\u9ed2",
+    "\u9ed9",
+    "\u9ee8",
+    "\u9f08",
+    "\u9f0e",
+    "\u9f13",
+    "\u9f20",
+    "\u9f3b",
+    "\u9f3e",
+    "\u9f62",
+    "\u9f63",
+    "\ufa1f",
+    "\ufa3f",
+    "\ufa55",
+    "\ufa5c",
+    "\ufa68"
+  ],
+  "mlp_dim": 1024,
+  "model_type": "metom",
+  "patch_size": 16,
+  "pool": "cls",
+  "transformers_version": "4.46.2"
+}

configuration_metom.py

@@ -0,0 +1,39 @@
+from typing import List
+
+from transformers import PretrainedConfig
+
+
+class MetomConfig(PretrainedConfig):
+    model_type = "metom"
+
+    def __init__(
+        self,
+        image_size: int = 128,
+        patch_size: int = 16,
+        labels: List[str] = [],
+        dim: int = 256,
+        depth: int = 6,
+        heads: int = 8,
+        mlp_dim: int = 1024,
+        pool: str = "cls",
+        channels: int = 3,
+        dim_head: int = 32,
+        dropout: float = 0.1,
+        emb_dropout: float = 0.1,
+        **kwargs
+    ):
+        super().__init__(
+            image_size=image_size,
+            patch_size=patch_size,
+            labels=labels,
+            dim=dim,
+            depth=depth,
+            heads=heads,
+            mlp_dim=mlp_dim,
+            pool=pool,
+            channels=channels,
+            dim_head=dim_head,
+            dropout=dropout,
+            emb_dropout=emb_dropout,
+            **kwargs
+        )

examples/README.md

@@ -0,0 +1,19 @@
+# Kuzushini Example Images
+Example Kuzushiji images stored in this directory are from [日本古典籍くずし字データセット](http://codh.rois.ac.jp/char-shape/).
+Note that the dataset is provided under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), so all images in the [examples/](./) directory are licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+Images in this directory are sampled from the test subset.
+
+![](./example1_4E00.jpg)
+*Class: U+4E00 (一)*
+
+![](./example2_5B9A.jpg)
+*Class: U+5B9A (定)*
+
+![](./example3_5009.jpg)
+*Class: U+5009 (倉)*
+
+## Reference
+[『日本古典籍くずし字データセット』](http://codh.rois.ac.jp/char-shape/)（国文研ほか所蔵／CODH加工） doi:10.20676/00000340
+
+提供：[ROIS-DS人文学オープンデータ共同利用センター](http://codh.rois.ac.jp/)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0485d6113b55737a62d1b518df32e81f6def519de31023a92d2fdfcd9f7187e6
+size 22587692

modeling_metom.py

@@ -0,0 +1,179 @@
+# This file is a modified version of the Vision Transformer - Pytorch implementation
+# https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/vit.py
+from typing import List, Union, Tuple
+
+from einops import rearrange, repeat
+from einops.layers.torch import Rearrange
+import torch
+from torch import nn
+from transformers import PreTrainedModel
+
+from .configuration_metom import MetomConfig
+
+
+try:
+    from flash_attn import flash_attn_func
+    FLASH_ATTENTION_2_AVAILABLE = True
+except ImportError:
+    FLASH_ATTENTION_2_AVAILABLE = False
+
+
+def size_pair(t):
+    return t if isinstance(t, tuple) else (t, t)
+
+
+class MetomFeedForward(nn.Module):
+    def __init__(self, dim, hidden_dim, dropout):
+        super().__init__()
+        self.net = nn.Sequential(
+            nn.LayerNorm(dim),
+            nn.Linear(dim, hidden_dim),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(hidden_dim, dim),
+            nn.Dropout(dropout)
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class MetomAttention(nn.Module):
+    def __init__(self, dim, heads, dim_head, dropout):
+        super().__init__()
+        inner_dim = dim_head *  heads
+        project_out = not (heads == 1 and dim_head == dim)
+        self.heads = heads
+        self.scale = dim_head ** -0.5
+        self.norm = nn.LayerNorm(dim)
+        self.attend = nn.Softmax(dim = -1)
+        self.dropout = nn.Dropout(dropout)
+        self.to_qkv = nn.Linear(dim, inner_dim * 3, bias = False)
+        self.to_out = nn.Sequential(
+            nn.Linear(inner_dim, dim),
+            nn.Dropout(dropout)
+        ) if project_out else nn.Identity()
+
+    def forward(self, x):
+        x = self.norm(x)
+        qkv = self.to_qkv(x).chunk(3, dim = -1)
+        q, k, v = map(lambda t: rearrange(t, "b n (h d) -> b h n d", h = self.heads), qkv)
+        dots = torch.matmul(q, k.transpose(-1, -2)) * self.scale
+        attn = self.attend(dots)
+        attn = self.dropout(attn)
+        out = torch.matmul(attn, v)
+        out = rearrange(out, "b h n d -> b n (h d)")
+        return self.to_out(out)
+
+
+class MetomSdpaAttention(MetomAttention):
+    def forward(self, x):
+        x = self.norm(x)
+        qkv = self.to_qkv(x).chunk(3, dim = -1)
+        q, k, v = map(lambda t: rearrange(t, "b n (h d) -> b h n d", h = self.heads), qkv)
+        out = nn.functional.scaled_dot_product_attention(q, k, v, dropout_p=self.dropout.p if self.training else 0.0)
+        out = rearrange(out, "b h n d -> b n (h d)")
+        return self.to_out(out)
+
+
+class MetomFlashAttention2(MetomAttention):
+    def forward(self, x):
+        x = self.norm(x)
+        qkv = self.to_qkv(x).chunk(3, dim = -1)
+        q, k, v = map(lambda t: rearrange(t, "b n (h d) -> b h n d", h = self.heads), qkv)
+        out = flash_attn_func(q, k, v, dropout_p=self.dropout.p if self.training else 0.0)
+        out = rearrange(out, "b h n d -> b n (h d)")
+        return self.to_out(out)
+
+
+class MetomTransformer(nn.Module):
+    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout, _attn_implementation = "eager"):
+        super().__init__()
+        if _attn_implementation == "flash_attention_2":
+            assert FLASH_ATTENTION_2_AVAILABLE, "FlashAttention-2 is not available. Please install `flash-attn`."
+        attn_cls = (
+            MetomAttention if _attn_implementation == "eager" else
+            MetomSdpaAttention if _attn_implementation == "sdpa" else
+            MetomFlashAttention2 if _attn_implementation == "flash_attention_2" else
+            MetomAttention
+        )
+        self.norm = nn.LayerNorm(dim)
+        self.layers = nn.ModuleList([])
+        for _ in range(depth):
+            self.layers.append(nn.ModuleList([
+                attn_cls(dim, heads = heads, dim_head = dim_head, dropout = dropout),
+                MetomFeedForward(dim, mlp_dim, dropout = dropout)
+            ]))
+
+    def forward(self, x):
+        for attn, ff in self.layers:
+            x = attn(x) + x
+            x = ff(x) + x
+        return self.norm(x)
+
+
+class MetomModel(PreTrainedModel):
+    config_class = MetomConfig
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+
+    def __init__(self, config: MetomConfig):
+        super().__init__(config)
+        image_height, image_width = size_pair(config.image_size)
+        patch_height, patch_width = size_pair(config.patch_size)
+        assert image_height % patch_height == 0 and image_width % patch_width == 0, "Image dimensions must be divisible by the patch size."
+
+        num_patches = (image_height // patch_height) * (image_width // patch_width)
+        patch_dim = config.channels * patch_height * patch_width
+        assert config.pool in {"cls", "mean"}, "pool type must be either cls (cls token) or mean (mean pooling)"
+        assert len(config.labels) > 0, "labels must be composed of at least one label"
+        assert config._attn_implementation in {"eager", "sdpa", "flash_attention_2"}, "Attention implementation must be either eager, sdpa or flash_attention_2"
+
+        self.to_patch_embedding = nn.Sequential(
+            Rearrange("b c (h p1) (w p2) -> b (h w) (p1 p2 c)", p1 = patch_height, p2 = patch_width),
+            nn.LayerNorm(patch_dim),
+            nn.Linear(patch_dim, config.dim),
+            nn.LayerNorm(config.dim),
+        )
+        self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, config.dim))
+        self.cls_token = nn.Parameter(torch.randn(1, 1, config.dim))
+        self.dropout = nn.Dropout(config.emb_dropout)
+        self.transformer = MetomTransformer(
+            config.dim, config.depth, config.heads, config.dim_head, config.mlp_dim, config.dropout, config._attn_implementation
+        )
+        self.pool = config.pool
+        self.to_latent = nn.Identity()
+        self.mlp_head = nn.Linear(config.dim, len(config.labels))
+        self.labels = config.labels
+
+    def forward(self, processed_image):
+        x = self.to_patch_embedding(processed_image)
+        b, n, _ = x.shape
+        cls_tokens = repeat(self.cls_token, "1 1 d -> b 1 d", b = b)
+        x = torch.cat((cls_tokens, x), dim=1)
+        x += self.pos_embedding[:, :(n + 1)]
+        x = self.dropout(x)
+        x = self.transformer(x)
+        x = x.mean(dim = 1) if self.pool == "mean" else x[:, 0]
+        x = self.to_latent(x)
+        return self.mlp_head(x)
+
+    def get_predictions(self, processed_image: torch.Tensor) -> List[str]:
+        logits = self(processed_image)
+        indices = torch.argmax(logits, dim=-1)
+        return [self.labels[i] for i in indices]
+
+    def get_topk_labels(
+        self, processed_image: torch.Tensor, k: int = 5, return_probs: bool = False
+    ) -> Union[List[List[str]], List[List[Tuple[str, float]]]]:
+        assert 0 < k <= len(self.labels), "k must be a positive integer less than or equal to the number of labels"
+        logits = self(processed_image)
+        probs = torch.softmax(logits, dim=-1)
+        topk_probs, topk_indices = torch.topk(probs, k, dim=-1)
+        topk_labels = [[self.labels[i] for i in ti] for ti in topk_indices]
+        if return_probs:
+            return [
+                [(label, prob.item()) for label, prob in zip(labels, probs)]
+                for labels, probs in zip(topk_labels, topk_probs)
+            ]
+        return topk_labels

preprocessor_config.json

@@ -0,0 +1,22 @@
+{
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "ViTImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 128,
+    "width": 128
+  }
+}