We have to talk about chatgpt

[Cola]

Alright, I'm sure this wil anger a lot of people, but we have to talk about chatGPT, as it is getting to a point where it may be useful.

I am currently at a pathology conference, and the ability of chatGPT(v4) to recognise highly specialized medical images is getting very good.

Everyone is very hesitant to actually use it of course, but very often it is on par with what a skilled pathologist would say about some histology slides.

 

So obviously I was curious how it would do on some nihonto related challenges.

I went ahead and copied a partial description of Aoi into the chatGPT prompt excluding the era, and then asked it for the era.

 

This was the answer:

 

Based on the detailed description of the sword, several clues suggest the time period it could belong to:

1. Blade Length and Curvature (Sori): A blade length of 73.8 cm with a curvature of 1.8 cm is typical of katana swords from the Edo period (1603–1868). During this period, katana lengths averaged around 70–80 cm, and the curvature was relatively shallow compared to earlier periods, like the Kamakura or Muromachi.

2. Number of Mekugi Holes: Three mekugi holes are relatively uncommon, but it's possible this blade has been remounted multiple times, which might suggest it was used over a long period or refitted during the Edo period, when swords were often reused and maintained.

3. Width at Base and Tip (Motohaba and Sakihaba): The blade's width at the base and tip seems consistent with those of later periods when swords were made more for practical use (e.g., Edo period and onward), rather than for the battlefield, where earlier swords were heavier and broader.

4. Jigane (Hada): The presence of clear itame hada (wood-grain pattern) can be found across many periods, but its clarity suggests a well-forged blade, which was a hallmark of higher craftsmanship during the late Muromachi (1336–1573) or early Edo periods.

5. Hamon: The described Gunome-midare hamon with deep nioikuchi and Ko-niedeki features are also found in swords from the Edo period. The midarekomi boshi style (where the hamon pattern continues into the tip) became more common in the Edo period.

Given these features, this sword likely originates from the early to mid-Edo period (17th century). During this time, swords were crafted more for personal defense and status, with less emphasis on heavy, large battlefield weapons and more on elegant, practical designs.

 

Very curious to hear from the experts whether this is all a complete hallucination, or whether it actually makes sense. Aoi art states this example is from the muromachi period btw.

 

Secondly, it can do image analysis as well now; I gave it only the picture of this sword (https://www.aoijapan.com/katana：bizen-koku-osafune-sukesada-sakunbthk-hozon-token/)

 

And it did a very  detailed description; but wrong on several points:

 

From the image, I can make some observations about the sword's structure and key features:

1. Blade Shape and Curvature (Sori): The blade appears to have a gentle curvature, typical of many Japanese swords from the Edo period. This curvature often reflects a design more oriented for personal defense rather than the more pronounced curves of earlier tachi blades used in mounted combat.

2. Hamon Pattern: The hamon, or temper line, appears to be midare (irregular wave-like pattern), potentially gunome-midare, as it shows undulating peaks. This style of hamon became quite popular in the Edo period and is a hallmark of high-quality swordsmithing, where differential heat treatment creates these distinct patterns along the cutting edge.

3. Nakago (Tang): The tang has two mekugi-ana (peg holes), which can indicate the age and re-mounting history of the sword. Multiple holes often mean the sword has been re-mounted in different handles or styles over time. The presence of a signature (mei) on the tang is a valuable clue, as it typically contains the name of the swordsmith and possibly the location or date of forging.

4. Kissaki (Tip): The tip of the sword looks well-defined with a noticeable boshi (the hamon continues onto the tip). This feature, depending on its shape and continuation of the hamon, is crucial for identifying the sword school or period.

5. Sugata (Overall Shape): The blade appears to have a standard shape typical of katana rather than tachi, with a balanced curve and medium-length tip. This style is also indicative of Edo period katana, used for practical wear rather than the longer and more curved blades used in earlier warfare.

 

All in all, I think it's quite interesting that it has apparently been trained on nihonto specific text, and is to some extent able to recognise details, although right now not very accurately it seems.  

Perhaps this can be useful for some of us; possibly also for translation tasks. 

[PNSSHOGUN]

It gives a sword dated 1572 to the Edo period, that says it all.....!

[GeorgeLuucas]

I've used chatGBT a few times in this hobby, in an attempt to clarify some vocabulary that I struggled to differentiate. (Example: difference between hakikake and kaen boshi)

I found it a little helpful... One thing I've noticed and worry about, is that it pulls from ALL internet information. Both reputable and less reputable. I've used it with mixed success, but it's given me some pretty misleading and incorrect information too. 

All in all, I've stopped using it entirely when studying Nihonto. I found it confused me more than it helped, and the time it took me to fact-check it's information; was the same amount of time it would take me to learn the info from the source anyway. If I stick with Markus Sesko publications, for example, I get a much better and more trustworthy understanding of what I am looking for.

A crazy new world! I wonder what AI will be capable of in just 10 years. 
Cheers,
-Sam

PS. Using it to look at photos is an interesting exercise. I wonder if it takes information from for-sale listings; we all know how accurate those can be 

Edited September 10 by GeorgeLuucas
PS.

[Lee Bray]

This came up in the cast tsuba thread by 'Dan Tsuba' a while ago. I forget the exact question but the AI fabricated several books and articles for its answer.

So no, I don't think it's close to reliable yet. If it learns from the internet, will it ever be?

[Shugyosha]

7 minutes ago, Lee Bray said:

 If it learns from the internet, will it ever be?

 

I have my doubts, it will boil down to the classic garbage in, garbage out scenario. The problem that may come around is that the water gets so muddied that it's no longer possible to rely on internet-based research as no-one knows what sources are trustworthy or that it becomes accepted as reliable because, well it's the internet isn't it. We are cursed with living in interesting times.

[Tcat]

IMHO ChatGPT is terrible at providing accurate information / facts and not good at true analysis of abstract information. Absolutely everything it tells you needs to be checked. Its strengths lie in quickly generating high volumes basic format content which can then be checked and refined by a human. Trust the 'facts' it spits out at your peril.

[Cola]

44 minutes ago, Lee Bray said:

This came up in the cast tsuba thread by 'Dan Tsuba' a while ago. I forget the exact question but the AI fabricated several books and articles for its answer.

So no, I don't think it's close to reliable yet. If it learns from the internet, will it ever be?

 

Which version of chatGPT did they use? For medical challenges as well, GPT3 was hallucinating up all kinds of fake references, but GPT4 has gotten a lot better in that regard. 

But we also have similar worries about whether the model recognises outdated or fraudulent publications as false. 

 

One option that seems to help a bit (I cannot check if this works for nihonto since I don't have the required books) is to input the books you trust in the model, and to let it answer questions based on those inputs. 

[Lee Bray]

1 hour ago, Cola said:

 

Which version of chatGPT did they use?

I don't know, it wasn't mentioned, but the post was dated 24th May, 2023, if that helps.
Page 10 on this thread.

 

[Cola]

2 hours ago, Lee Bray said:

I don't know, it wasn't mentioned, but the post was dated 24th May, 2023, if that helps.
Page 10 on this thread.

 

That's got to be GPT3 then GPT4 is much more recent; and since he put the full convo in there I can just recreate it with GPT4; it's not very concise though.

 

Did Japan produce cast-iron ingots for commercial trade during the Edo period?

 

Japan did not widely produce cast-iron ingots for commercial trade during the Edo period (1603–1868). The Edo period, under the Tokugawa shogunate, was marked by a policy of sakoku (national isolation), which limited foreign trade and interaction. Japan did have a robust ironworking tradition during this time, but the production and trade of cast iron were primarily for domestic use, not for large-scale export.

Here’s a more detailed look at the situation during the Edo period:

Domestic Iron Production:

• Iron Production Techniques: The Japanese iron industry was based on traditional methods like tatara, a smelting process that involved producing steel (including tamahagane) from iron sand (satetsu). This method was labor-intensive and produced relatively small quantities of high-quality steel, primarily used for tools, weapons (e.g., nihonto), and agricultural implements. Cast iron, in contrast, is more brittle and less suitable for items like swords.

• Domestic Use of Cast Iron: Although Japan produced cast iron in small amounts, it was used mostly for specific items like pots, pans, kettles (e.g., tetsubin), and some larger construction or engineering components. Cast iron wasn’t considered a premium material, so its application was limited compared to wrought iron and steel, which were more highly valued for their durability and workability.

Commercial Trade and Sakoku Policy:

• Limited Trade: Under the sakoku policy, Japan only traded with a few nations, primarily China, Korea, and the Netherlands (through the Dutch East India Company in Nagasaki). The scope of this trade was tightly controlled, and there is no significant evidence of large-scale exports of cast-iron ingots during this period. The iron trade primarily focused on domestic demand rather than export markets.

• Iron Imports: Japan actually imported iron ore and other raw materials during the Edo period, primarily from China and Korea, to supplement its own iron resources. This further indicates that the country was not in a position to produce surplus cast-iron ingots for export.

Why No Large-Scale Cast-Iron Export?

1. Economic Priorities: The Tokugawa shogunate prioritized self-sufficiency and maintaining domestic production to support agricultural and artisan communities. Iron production was focused on fulfilling internal needs, especially for farming tools and construction materials, rather than generating export surpluses.

2. Technical Limitations: Japan's tatara smelting technique, while excellent for producing steel, was not geared toward mass-producing cast iron in ingot form for trade. Other nations, particularly in Europe and China, had developed larger-scale iron foundries that could mass-produce cast iron more efficiently.

In summary, Japan’s production of cast iron during the Edo period was limited to meeting domestic needs, and it was not a major player in the international cast-iron trade during this time.

 

Provide references for these statements:

 

During the Edo period (1603-1868), Japan was known for its production and trade of various iron products, including cast iron. While Japan is not particularly rich in natural iron resources, it developed a significant iron industry, especially in regions like Mimasaka, Bizen, and Bitchū, where iron was refined and made into products like tools, swords, and other artifacts. Some sources mention that iron was cast into ingots and traded domestically, though Japan's primary focus was on the refinement of iron sands into steel for the production of swords and armory(

Gunbai: Ancient Japanese Warfare)(Gunbai: Ancient Japanese Warfare).

Commercial export of iron was limited, primarily because Japan's isolationist policy (Sakoku) restricted foreign trade to a few select ports. However, Japanese merchants did engage in limited foreign trade, particularly with the Dutch and Chinese, who exchanged iron and copper. Japanese cast iron foundries, particularly in regions like Oshu, specialized in producing high-quality iron products for both domestic use and trade(Boonie Hicks)(Gunbai: Ancient Japanese Warfare).

 

Provide references about the production of cast-iron ingots in Japan during the Edo period

 

The production of cast-iron ingots in Japan during the Edo period (1603–1868) was not a major commercial focus. Japan primarily produced iron for domestic consumption, with iron sand (satetsu) refined into higher-quality steel for tools, weapons, and other local uses. Regions like Mimasaka, Bizen, and Bitchū were known for iron production, though large-scale ingot production for international trade was uncommon due to the country’s isolationist policy (sakoku)(

Gunbai: Ancient Japanese Warfare)(Gunbai: Ancient Japanese Warfare).

Iron was smelted using traditional techniques like tatara smelting, which primarily aimed at producing high-quality steel (like tamahagane) rather than mass production of cast iron. The limited cast-iron products, such as kettles and other domestic wares, were largely for local use. Even though some cast iron was produced, it wasn't part of large-scale commercial trade, especially since Japan imported iron ore from China and Korea to supplement its own supplies(Boonie Hicks).

For further insight into traditional Japanese iron production, including casting, the regional foundries like those in Oshu are still known for their historical contributions to the production of high-quality cast iron(Boonie Hicks)(Gunbai: Ancient Japanese Warfare).

 

[GeorgeLuucas]

https://www.militaria.co.za/nmb/topic/47942-ai-and-nihonto-chatgbt/#comment-497930

Here's a thread I started in the Izakaya in February about ChatGBT, with the prompts and answers that I fed it. 
Cheers,
-Sam

[George KN]

I'm not a nihonto expert, but I work with AI language and image models regularly as part of my job and hobbies, and there's a lot of misinformation about what AI can and can't do...

 

Language models do not know things. They simply create text that reads like the most likely answer based on the previous context it has access too.

 

This means it is very good at creating things that sound right, but not things that are accurate, or would disagree with its previous context. 

 

Yes, you could create a more accurate model using only reputable sources (e.g. adding only Sesko's and others work, giving it none of the mess that is online sources (this would be stealing without permission though)). But it has no ability to reason or solve conflicts of information - it would simply try to create what best fitted the theme of the works previously provided, leading to decent accuracy but regular terrible mistakes that even a beginner would catch.

 

This isn't to say that AI can't be used in the field of Nihonto though - I actually have high hopes with enough images and sources image models would be very useful at matching signatures against indexed swordsmiths. Similarly, the initial terribly inaccurate assessment can be used to speed up writing a human analysis or text, as it at least provides a starting point, however wrong.

 

In general AI like most tech tools thrive on very specific datasets and tasks - general use (like trying to analyze all things about a sword), not so much.

[John C]

Just my two cents - I think AI can eventually be useful in Nihonto identification, however similar to the way it is trained to spot cancer, known valid examples will have to be offered so it can "learn" what to look for. In breast cancer, for example, thousands of scans that were identified by the doctor as normal but where cancer was later identified in the individual, were used to train the AI. Currently, it can identify cancer not identifiable to the radiology technician (I forget what the accuracy rate is).

I think an area where it could be most useful would be in mei authentication.  If it can be shown enough scans of legitimate mei for a given smith, it could learn the subtleties of his chisel marks or whatever. Access to this program could be used by shinsa or by sword buyers just wanting to confirm a signature. 

The problem will always be garbage in; garbage out. There would need to be certainty that the mei used to train the AI are legitimate.

 

John C.

 

[Bazza]

Just a 'simples' question - does any know or care to guess at the carbon footprint of chatGBT???  Or the internet???

 

BaZZa.

[Lee Bray]

Rather large, Bazza. Knew it was big but just googled and they reckon it uses the same amount of power as Sweden currently and will rise to Germany's supply by 2026, roughly 500 terawatt hours rising to 1000 terawatt hours. That's AI alone, no internet.

 

 

[Shugyosha]

If you add in Bitcoin and the other crypto-currencies, it starts to get a bit scary:

 

https://www.imf.org/...-tax-policy-can-help

[raynor]

Speaking as someone with an education in the IT field - Large language models like ChatGPT are getting more accurate every day, but I firmly place them within a rule as old as the internet: Do not take serious medical or academic advice from an online generated source, chatGPT and its like should be treated as a fancy version of wikipedia, nothing more.

[Cola]

This one was too funny to keep from you:

[Jussi Ekholm]

I have pretty much zero knowledge on AI but I would think in the future the possibilities are going way above my head. However on personal level I just love researching stuff, I will keep doing the same thing even though AI will at some point surpass my own capabilities. Even at the current level, if for example AI would have access all the reference books I have in my personal library it could provide amazing info. Bit scary to even think about what it would be able to do with so much high quality information and data.

[Lewis B]

AI has a lot to learn. In my efforts to understand swordsmiths who combine both Yamato and Soshuden characteristics this is what ChatGPT has decided I need to know..... back to the books

 

Here is an artistic rendering of a nihonto sword blending both Yamato and Soshu characteristics, as described. It captures the elegance and robustness of both styles. ....

[raynor]

A large part in how these language model interfaces (chatgpt and similar) works is that a large part of the training is done by humans, not just the massive calculations carried out by data centers. Actual humans sit down and give feedback on the answers the bots put out, so that they and their creators learn what language carries best when giving feedback to users (customers).

 

Here is a experiment I think members of this forum is rather uniquely qualified to carry out - many members here are a lot more knowledgeable about nihonto then the average Joe. Ask chatgpt questions with variable degrees of difficulty on nihonto or any field you are confident in and you will see it is correct on many things, but also incorrect. When incorrect it will, until corrected speak, in other words "lie" with the utmost confidence worthy of any politician. This is one of the problems, for most people the answers about nihonto when presented with confidence by the allmighty AI will in many cases pass as facts, truth.

 

This of course applies to any other field not just nihonto, wherein lie a tremendous challenge unless these systems are subjected to quality control and regulations.

[Bruce Pennington]

9 hours ago, raynor said:

Actual humans sit down and give feedback on the answe

Is that something WE can do as we "chat" with it?  Or is that something that is set in stone before the bot is released into the wild?

[Pincheck]

9 hours ago, Bruce Pennington said:

Is that something WE can do as we "chat" with it?  Or is that something that is set in stone before the bot is released into the wild?

ChatGPT keeps the chat history in its contextual memory so you could feed it information and then ask it questions about what you’ve told it. That history is gone when you start a new chat though. But you can train a custom GPT and save it so other can use it. There are input limits though, I think I read something like 32k tokens, a token is a word fragment like “ing” or “tion”. AI is generally good at things that are incontrovertible mathematical facts, like computer code, and really bad at woolly opinion based things with vague and inconsistent descriptions, like kantei.

[Gakusee]

A few of us have been training ChatGPT on Nihonto translations for the last couple of years. The results are very good. 
 

What people need to realise is that this is not AI as in autonomous thinking entity. These are just linguistic models with some inference and summarisation ability, but that is it. They can crunch through publications and summarise them eloquently for less esoteric subjects. However for arcane subjects like Nihonto, your best bet is only translations for now. ChatGPT still gets smith and location names wrong but the basic terms it has mastered pretty well now. 

[vajo]

Don't use A.I.

 

A.I. makes you lazy and fool.

No one will end up with the answer of an A.I. question.

 

A.I. is the end of thinking.

[Gakusee]

13 minutes ago, vajo said:

Don't use A.I.

 

A.I. makes you lazy and fool.

No one will end up with the answer of an A.I. question.

 

A.I. is the end of thinking.

Chris, I do not agree that AI will just yet stop us from thinking. Quite the opposite, we need to carefully read what it has written, analyse it, dissect and question it to ensure we agree with what is has written and whether it is true. The software models sometimes invent statements, which look very credible and persuasive, but are in fact figments of its creativity. 
It should be viewed as a good librarian which can point us to good sources of information and summarise these at a high level of abstraction. 
it is not substitute for own experience and knowledge but often produces much more thorough and informative results than a Google or Bing request. The important thing is that you can modulate the interaction and specify how it should give it the answer: verbatim, summarised or with interpretation of what it might mean. I have used all of these as sometimes it summarises things while I need the full, precise detail. 
 

it could remove mundane and laborious tasks of looking up kanji or esoteric meanings but is not there yet. I think it needs another 3-5 years to be more useful. 

[raynor]

I disagree that AI is the end of thinking, but together with say autotune it will be a heavy blow to creativity.

 

It will however be hard to catch people cheating in academia now, until they have to actually do the job. Hopefully many steps from the operating table...

 

17 hours ago, Bruce Pennington said:

Is that something WE can do as we "chat" with it?  Or is that something that is set in stone before the bot is released into the wild?

This varies through models, but it is a shared path - models like chatgtp get a lot of free training through user usage, but they also do a lot of in house specific feedback training by personnel so the answer is yes and no depending on the model.

[vajo]

Look at all the fake a.i. pictures people belive they are true. 

How should they proof if a text is true when they ask the a.i. 

 

[raynor]

This is exactly why it needs laws and regulation, I rarely agree with Elon Musk on much, but he is spot on with how urgent it is. The technology is too impactful to just be left for companies seeking profit.

[Bruce Pennington]

Hey, who needs A.I. - I can prove Abe Lincoln shot JFK.  I have a photo of him with the rifle!

[Kantaro]

The pope arrived in Belgium yesterday and A.I. has pics...