• Knock_Knock_Lemmy_In@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    1
    ·
    1 year ago

    Run something with a 70% failure rate 10x and you get to a cumulative 98% pass rate. LLMs don’t get tired and they can be run in parallel.

    • jsomae@lemmy.ml
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 year ago

      The problem is they are not i.i.d., so this doesn’t really work. It works a bit, which is in my opinion why chain-of-thought is effective (it gives the LLM a chance to posit a couple answers first). However, we’re already looking at “agents,” so they’re probably already doing chain-of-thought.

      • Knock_Knock_Lemmy_In@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        Very fair comment. In my experience even increasing the temperature you get stuck in local minimums

        I was just trying to illustrate how 70% failure rates can still be useful.

      • jsomae@lemmy.ml
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        I’m impressed you can make strides with Rust with AI. I am in a similar boat, except I’ve found LLMs are terrible with Rust.

          • 𝕛𝕨𝕞-𝕕𝕖𝕧@lemmy.dbzer0.comdeleted by creator
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            i find that rust’s architecture and design decisions give the LLM quite good guardrails and kind of keep it from doing anything too wonky. the issue arises in cases like these where the rust ecosystem is quite young and documentation/instruction can be poor, even for a human developer.

            i think rust actually is quite well suited to agentic development workflows, it just needs to mature more.