Molmo

The Molmo class provides core functionality for this module.

class Molmo()

use_local · boolean · default: "False"

If True, inference call is run on the local VM, else offloaded onto GRID-Cortex. Defaults to False.

def run()

image · np.ndarray · required

The input RGB image of shape $(M,N,3)$.

prompt · str · required

The question to answer about the media.

Returns · str

The response to the prompt.

Inference Call

from grid.model.perception.vlm.molmo import Molmo
car = AirGenCar()

# We will be capturing an image from the AirGen simulator 
# and run model inference on it.

img =  car.getImage("front_center", "rgb").data

model = Molmo(use_local = False)
result = model.run(image=img, prompt=<prompt>)
print(result)

License

This code is licensed under the Apache 2.0 License.

Source

https://github.com/allenai/molmo