Error while running locally using run-cli.sh

[ved-asole]

[ERROR] No plugin found for prefix 'jlama-core' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/ved/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1]
org.apache.maven.plugin.prefix.NoPluginFoundForPrefixException: No plugin found for prefix 'jlama-core' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/ved/.m2/repository), central (https://repo.maven.apache.org/maven2)]

[tjake]

Hi @ved-asole can you give some more details like what you did to run this? did you just checkout the code and run-cli.sh?

[ved-asole]

As per the documentation I used the below commands :

./run-cli.sh download gpt2-medium
./run-cli.sh download -t XXXXXXXX meta-llama/Llama-2-7b-chat-hf
./run-cli.sh download intfloat/e5-small-v2

[tjake]

Thanks, I recently added support for multiple jvms which messed this up. I'll fix. In the meantime use the code from the latest release tag git checkout v0.2.1 and it should work

[ved-asole]

Thanks for the update. I will use the older v0.2.1 version for now

[ved-asole]

@tjake I'm not sure where I can get the older v0.2.1 version/ Could you please suggest how i can get it?

[ved-asole]

Getting below error while running v0.2.1 :

Command : sudo ./run-cli.sh download gpt2-medium

An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "make" (in directory "/home/ved/Downloads/temp/Jlama-0.2.1/jlama-native"): error=2, No such file or directory
[ERROR] around Ant part ...<exec resolveexecutable="true" failonerror="true" executable="make">... @ 4:71 in /home/ved/Downloads/temp/Jlama-0.2.1/jlama-native/target/antrun/build-main.xml

[tjake]

You need to build the native extensions requires make and gcc.

I realize this is all a PITA. I'm going to add release artifacts for all the platforms when I get a chance.

[tjake]

The easiest way to "just run a model" is with langchain4j integrations. Try this: https://github.com/langchain4j/langchain4j-examples/tree/main/jlama-examples

[ved-asole]

I want to run the models locally. Will it run with the langchain4j implementations?

[tjake]

Yes, it uses the maven artifacts which are pre-built

[ved-asole]

I got the below error while using it :

2024-09-02T15:41:34.997+05:30  INFO 94027 --- [pdf-assistant] [           main] c.g.t.j.safetensors.SafeTensorSupport    : Downloaded file: /home/ved/.jlama/models/TinyLlama-1.1B-Chat-v1.0-Jlama-Q4/tokenizer_config.json
2024-09-02T15:41:44.526+05:30  WARN 94027 --- [pdf-assistant] [           main] c.github.tjake.jlama.util.MachineSpec    : Unknown vector type: 128
2024-09-02T15:41:44.612+05:30  WARN 94027 --- [pdf-assistant] [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'chatController' defined in file [/mnt/hdd1/Coding/springWorks/pdf-assistant/target/classes/com/vedasole/pdf_assistant/ChatController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
2024-09-02T15:41:44.615+05:30  INFO 94027 --- [pdf-assistant] [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2024-09-02T15:41:44.638+05:30  INFO 94027 --- [pdf-assistant] [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-09-02T15:41:44.670+05:30 ERROR 94027 --- [pdf-assistant] [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'chatController' defined in file [/mnt/hdd1/Coding/springWorks/pdf-assistant/target/classes/com/vedasole/pdf_assistant/ChatController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:795) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:237) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1375) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1212) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:971) ~[spring-context-6.1.12.jar:6.1.12]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) ~[spring-context-6.1.12.jar:6.1.12]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.3.3.jar:3.3.3]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.3.3.jar:3.3.3]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.3.3.jar:3.3.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) ~[spring-boot-3.3.3.jar:3.3.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) ~[spring-boot-3.3.3.jar:3.3.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) ~[spring-boot-3.3.3.jar:3.3.3]
	at com.vedasole.pdf_assistant.PdfAssistantApplication.main(PdfAssistantApplication.java:44) ~[classes/:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) ~[spring-beans-6.1.12.jar:6.1.12]
	... 19 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:178) ~[spring-beans-6.1.12.jar:6.1.12]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.12.jar:6.1.12]
	... 33 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:161) ~[jlama-core-0.2.1.jar:na]
	at dev.langchain4j.model.jlama.JlamaModel$Loader.load(JlamaModel.java:92) ~[langchain4j-jlama-0.33.0.jar:na]
	at dev.langchain4j.model.jlama.JlamaChatModel.<init>(JlamaChatModel.java:51) ~[langchain4j-jlama-0.33.0.jar:na]
	at dev.langchain4j.model.jlama.JlamaChatModel$JlamaChatModelBuilder.build(JlamaChatModel.java:29) ~[langchain4j-jlama-0.33.0.jar:na]
	at com.vedasole.pdf_assistant.PdfAssistantConfig.conversationalRetrievalChain(PdfAssistantConfig.java:56) ~[classes/:na]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) ~[spring-beans-6.1.12.jar:6.1.12]
	... 34 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[na:na]
	at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:153) ~[jlama-core-0.2.1.jar:na]
	... 41 common frames omitted
Caused by: java.lang.UnsupportedOperationException: null
	at com.github.tjake.jlama.tensor.operations.PanamaTensorOperations.quantize(PanamaTensorOperations.java:1484) ~[jlama-core-0.2.1.jar:na]
	at com.github.tjake.jlama.tensor.operations.NativeTensorOperations.quantize(NativeTensorOperations.java:355) ~[jlama-native-0.2.1-linux-x86_64.jar:na]
	at com.github.tjake.jlama.model.AbstractModel.<init>(AbstractModel.java:117) ~[jlama-core-0.2.1.jar:na]
	at com.github.tjake.jlama.model.llama.LlamaModel.<init>(LlamaModel.java:55) ~[jlama-core-0.2.1.jar:na]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[na:na]
	... 44 common frames omitted

[tjake]

Which model are you loading and what kind of cpu are you loading it on?

[tjake]

Looking at the line it seems to be a non-arm AVX which I can support but currently needs a code change. I can fix that.