At this point, you could let the user select one of the search results as a new visual query image and further refine the search.
No search tool is perfect, and visual search is no exception. Sometimes the content of an image will be so obscured or so dependent on human semantic understanding that a visual search will turn up with few relevant results. However, when used with clear images and good search parameters, the eVe search results are usually quite accurate. When combined with a text search, visual search results can be absolutely astounding.
Although you've seen a very simple example in this article, more advanced applications are easy to create. Perhaps the most complex part is coming up with a graphical user interface that makes it clear to the user how to select objects and how to best search visually. No doubt, visual search interfaces will improve as visual search appears in more and more venues.