advertisement


Mac OSX Spotlight re-indexing external USB HDD upon every boot

ToTo Man

the band not the dog
This is the first time I've encountered this issue in my almost two decades of using Mac OSX and I don't know what to do...

Every time I boot up one of my recently acquired 2012 Mac Minis, onto which I did a clean install of El Capitan, Spotlight spends the first half hour or so re-indexing my external HDD. I thought nothing of it for the first few days, I even left the machine idle for a whole day so that Spotlight could do its thing, so it's not as if I haven't given it chance to index the drive. But I'm a month in now and it still does it on every startup, even if I haven't added or edited files on the drive.

I read a suggestion to add the affected drive to the privacy tab in Spotlight preferences and then remove it to force a re-index. I tried this but it doesn't seem to have helped. I read on another Apple discussion that replacing the Logic board fixed a similar problem on a MacBook Pro, which seems a drastic course of action!

Is there anything else I can try that's less invasive? I could resort to putting the drive permanently onto Spotlight's privacy list, but then I would presumably be unable to search for files, which would be somewhat inconvenient! I also worry that, if Spotlight is constantly re-indexing my external HDD on every boot, is it doing the same with the internal SSD, and therefore causing unnecessary wear and tear?
 
It is worth noting what format has been applied to the external drive. None Apple formats such as FAT32 and exFAT tend to have their index located on the Mac system drive within /.Spotlight-V100 in file .store.db. As these drives are assumed to be also plugged into other machines, Spotlight must refresh the index to account for activity that happens elsewhere. I suspect that the trigger for this has always been a bit wonky, such as in your case via a simple system start/restart.

Typically, if my external is only to be used with my Macs, I'll make sure to format it to OS X Extended (Journaled) (aka HFS+ Journaled). The only downside being that these won't be accessible on Windows machines (at least, not without third party software such as 'MacDrive').

Having reported the above, typically, the main Spotlight related issue with FAT32/exFAT drives is that indexing need be force activated, as it isn't always switched on by default for these.

Regardless, as your's is obviously indexing, suggest using 'First Aid' on the external drive via 'Disk Utility' and then repeat the add/remove drive via Spotlight preference's privacy list.

Alternatively, the ultimate Spotlight reset is to erase the entire metadata store and then reindex all using the following series of 'mdutil' commands via 'Terminal'.

sudo mdutil -Ea
sudo mdutil -ai off
sudo mdutil -ai on


where
sudo Executes a command as another user (in this case 'root').
mdutil Manages the metadata stores used by Spotlight.
-E Causes each local store for the volumes indicated to be erased. The stores will be rebuilt if appropriate.
-a Applies command to all volumes.
-i on | off Set the indexing status for the provided volumes to on or off.​

This will take some time and should be allowed to complete without shutdown or sleep interruptions.

If all else fails, suggest temporarily moving the data off HDD, reformatting to OS X Extended (Journaled) and then moving the data back on.
 
Last edited:
It is worth noting what format has been applied to the external drive. None Apple formats such as FAT32 and exFAT tend to have their index located on the Mac system drive within /.Spotlight-V100 in file .store.db. As these drives are assumed to be also plugged into other machines, Spotlight must refresh the index to account for activity that happens elsewhere. I suspect that the trigger for this has always been a bit wonky, such as in your case via a simple system start/restart.

Typically, if my external is only to be used with my Macs, I'll make sure to format it to OS X Extended (Journaled) (aka HFS+ Journaled). The only downside being that these won't be accessible on Windows machines (at least, not without third party software such as 'MacDrive').

Having reported the above, typically, the main Spotlight related issue with FAT32/exFAT drives is that indexing need be force activated, as it isn't always switched on by default for these.

Regardless, as your's is obviously indexing, suggest using 'First Aid' on the external drive via 'Disk Utility' and then repeat the add/remove drive via Spotlight preference's privacy list.

Alternatively, the ultimate Spotlight reset is to erase the entire metadata store and then reindex all using the following series of 'mdutil' commands via 'Terminal'.

sudo mdutil -Ea
sudo mdutil -ai off
sudo mdutil -ai on


where
sudo Executes a command as another user (in this case 'root').
mdutil Manages the metadata stores used by Spotlight.
-E Causes each local store for the volumes indicated to be erased. The stores will be rebuilt if appropriate.
-a Applies command to all volumes.
-i on | off Set the indexing status for the provided volumes to on or off.​

This will take some time and should be allowed to complete without shutdown or sleep interruptions.

If all else fails, suggest temporarily moving the data off HDD, reformatting to OS X Extended (Journaled) and then moving the data back on.
Or, alternatively, get something which just works and isn't a PITA.
What springs to mind? Well not a ****ing mac for starters.
 
Or, alternatively, get something which just works and isn't a PITA.
What springs to mind? Well not a ****ing mac for starters.
Yes, those once every two decades issues can be crippling.

At least with Windows, one gets used to dealing with issues on a more regular basis.

Then there is the moving target of the infinite number of Linux iterations, where one must deal with problems all up front, only to find that any shared potential solutions are out of date before the virtual ink is dry on the GNU license.
 
Or, alternatively, get something which just works and isn't a PITA.
What springs to mind? Well not a ****ing mac for starters.
Well, I've got a MacBook running Catastrophia or somesuch...
And I've got a PC running W7.
Guess which is which?
I know what I'm talking about.
 
Well, I've got a MacBook running Catastrophia or somesuch...
And I've got a PC running W7.
Guess which is which?
I know what I'm talking about.
I love guessing games. Ah, let's see...W7 is the one that reached end of life back in January 2020, right?
 
Well, I've got a MacBook running Catastrophia or somesuch...
And I've got a PC running W7.
Guess which is which?
I know what I'm talking about.
From what I see none of you know what you are talking about. Another case of the 'blind leading the blind' or maybe yet again 'a little knowledge is a dangerous thing'.

What you have are three O/S each of which is aimed at a different audience.

I use all 3 daily.

Oh you mean Crapolina? That is Apples Vista moment.

Cheers,

DV
 
From what I see none of you know what you are talking about. Another case of the 'blind leading the blind' or maybe yet again 'a little knowledge is a dangerous thing'.

What you have are three O/S each of which is aimed at a different audience.

I use all 3 daily.

Cheers,

DV
Yes, we know, you live in a Linux VM that is running within a Windows VM that is running within a Windows partition on a Mac, right?

Is is dark in there?
 
Yes, you live in a Linux VM that is running within a Windows VM that is running within a Windows partition on a Mac, right?

Is is dark in there?
Wrong!

I have several machines some are Apple hardware and others well are not Apple.

I support all three platforms in my local community for free since I am retired, have the time, and love the challenge of computers whether micro based or mainframes.

I have been able to clone Big Sur via a standard USB 2 or 3 port onto an SSD and shown that this SSD will boot from several Apple machines from 2014 to 2017. The CCC people claim that this is not possible. Check out the Internet as I may be out of date by now.

You have no idea of my technical background but as ever if you can look something up on the Internet you become an instant expert - no? some of us really are experts.........

Have fun,

DV
 
Wrong!

I have several machines some are Apple hardware and others well are not Apple.

I support all three platforms in my local community for free since I am retired, have the time, and love the challenge of computers whether micro based or mainframes.

I have been able to clone Big Sur via a standard USB 2 or 3 port onto an SSD and shown that this SSD will boot from several Apple machines from 2014 to 2017. The CCC people claim that this is not possible. Check out the Internet as I may be out of date by now.

You have no idea of my technical background but as ever if you can look something up on the Internet you become an instant expert - no? some of us really are experts.........

Have fun,

DV
I would never presume to doubt your technical background, sir. Nor should you, mine.

You did neglect to mention your expertise when it comes to habitually slagging off others' advice on here, though.
 
Yes, we know, you live in a Linux VM that is running within a Windows VM that is running within a Windows partition on a Mac, right?

Is is dark in there?
Wow, you must be very intellectually agile to be able to make jokes like that. I imagine your wit makes you very popular with your friend. If you have one. Do you get out much?
So back to wonderful Macs...E ****ing G, I have to "eject" USB storage. That's so much better than a PC, where I just airily pull out (hey Beavis, he said pull out). I like that the mac cares enough to give me a little telling off message if I forget and treat it like it's grown up, er a PC, and pull the storage out anyway. Then makes me have to say sorry and perform obeisances if I re connect.
Want to run a third party program on a mac? Better just hope there's a fourth party cludge up available to make it work.
And getting fed up with how speedy and efficient it is? Running too many programs? Just keep on installing those updates.
Okay, different audience. I use a computer as a tool, when I'm not just spaffing about on the net. Call me old fashioned, but I like tools that:
Don't require constant dicking about with and their egos massaging
Do what they're supposed to do, efficiently and ergonomically
Work with other stuff first time, like printers, displays, hard drives
Are repairable and don't have to get chucked away when what would just be a card replacement type failure on a pc happens.
You might like fooling around with computers for its own sake - from what you posted above, I apprehend you do - but for me, well, a means to an end.
 
It is worth noting what format has been applied to the external drive. None Apple formats such as FAT32 and exFAT tend to have their index located on the Mac system drive within /.Spotlight-V100 in file .store.db. As these drives are assumed to be also plugged into other machines, Spotlight must refresh the index to account for activity that happens elsewhere. I suspect that the trigger for this has always been a bit wonky, such as in your case via a simple system start/restart.

Typically, if my external is only to be used with my Macs, I'll make sure to format it to OS X Extended (Journaled) (aka HFS+ Journaled). The only downside being that these won't be accessible on Windows machines (at least, not without third party software such as 'MacDrive').

Having reported the above, typically, the main Spotlight related issue with FAT32/exFAT drives is that indexing need be force activated, as it isn't always switched on by default for these.

Regardless, as your's is obviously indexing, suggest using 'First Aid' on the external drive via 'Disk Utility' and then repeat the add/remove drive via Spotlight preference's privacy list.

Alternatively, the ultimate Spotlight reset is to erase the entire metadata store and then reindex all using the following series of 'mdutil' commands via 'Terminal'.

sudo mdutil -Ea
sudo mdutil -ai off
sudo mdutil -ai on


where
sudo Executes a command as another user (in this case 'root').
mdutil Manages the metadata stores used by Spotlight.
-E Causes each local store for the volumes indicated to be erased. The stores will be rebuilt if appropriate.
-a Applies command to all volumes.
-i on | off Set the indexing status for the provided volumes to on or off.​

This will take some time and should be allowed to complete without shutdown or sleep interruptions.

If all else fails, suggest temporarily moving the data off HDD, reformatting to OS X Extended (Journaled) and then moving the data back on.
Thanks for the detailed reply, Craig.

I should have noted in my OP that the external drive is already formatted OSX Extended (Journaled). This was done in 2015 on a Mac Pro using Disk Utility in El Capitan. The external drive was then connected to a 2010 Mac Mini where it performed without issue until I replaced the 2010 Mini with a 2012 Mini a few weeks ago.

Also, I notice I made a typo in my OP, I wrote USB but I actually have the drive connected by FireWire, if that makes a difference? (I'm assuming not as it was connected to my Mac Pro and 2010 Mini by FireWire without issue).

I ran First Aid on the disk a few weeks ago (I did this as I thought the constant read/write activity was a signal of imminent failure before I discovered it was Spotlight causing the activity). No faults were found then, but I shall run it again and then repeat the adding/removal of the drive from Spotlight Privacy.

If that doesn't work then I'll resort to the Terminal commands.

PS - Would Spotlight's misbehaving have anything to do with the fact that the external drive is rammed full of data with very little free space available (it's a 6TB drive with only 6GB free space left on it)?
 
Thanks for the detailed reply, Craig.

I should have noted in my OP that the external drive is already formatted OSX Extended (Journaled). This was done in 2015 on a Mac Pro using Disk Utility in El Capitan. The external drive was then connected to a 2010 Mac Mini where it performed without issue until I replaced the 2010 Mini with a 2012 Mini a few weeks ago.

Also, I notice I made a typo in my OP, I wrote USB but I actually have the drive connected by FireWire, if that makes a difference? (I'm assuming not as it was connected to my Mac Pro and 2010 Mini by FireWire without issue).

I ran First Aid on the disk a few weeks ago (I did this as I thought the constant read/write activity was a signal of imminent failure before I discovered it was Spotlight causing the activity). No faults were found then, but I shall run it again and then repeat the adding/removal of the drive from Spotlight Privacy.

If that doesn't work then I'll resort to the Terminal commands.

PS - Would Spotlight's misbehaving have anything to do with the fact that the external drive is rammed full of data with very little free space available (it's a 6TB drive with only 6GB free space left on it)?
You're welcome, @ToTo Man.

6GB would be scant few if this were a system drive, however, for strictly file storage it is plenty of free space.

I can provide a script that would run the three terminal commands in succession, if you wish. You could let them go overnight without any intervention.
 
You're welcome, @ToTo Man.

6GB would be scant few if this were a system drive, however, for strictly file storage it is plenty of free space.

I can provide a script that would run the three terminal commands in succession, if you wish. You could let them go overnight without any intervention.
That would be good Craig, thanks. I'm off to bed now and have already shut down the Mini, but I'll be able to run them tomorrow.
 
PS - Would Spotlight's misbehaving have anything to do with the fact that the external drive is rammed full of data with very little free space available (it's a 6TB drive with only 6GB free space left on it)?

Quite possibly. There are likely many fragmented files with that little free space, so with a spinning HD you're spending a lot of time moving between them rather than indexing them. Do any of these files change on a daily basis?
 
Quite possibly. There are likely many fragmented files with that little free space, so with a spinning HD you're spending a lot of time moving between them rather than indexing them. Do any of these files change on a daily basis?
With 6GB of free space there will be ample room for /.Spotlight-V100/Store-V2/<UUID> folder(s), as these contain relatively large text files which are still quite small in size relative to other files.

For example, connected to my GF's iMac here is a 750GB USB HDD for Time Machine backups. I've set her Time Machine to exclude iTunes Libraries as she has a very large photos library file that takes precedence over all else here (plus the Mac has a 1TB HDD vs. the 750MB USB, the latter a castoff of mine). I've just checked now and the <UUID> folder on the external is 1.2MB in size, this despite Spotlight having been at indexing this volume since she purchased the Mac back in 2012 (thankfully, not continuously the whole time).

I do see what you mean wrt normal file content vs. fragmentation, but Spotlight indexing appears to be quite robust when it comes to doing its job; that is, other than when a metadata file becomes corrupted, ironically, something that can easily happen when such as @ff1d1l decides to pull out 'Butt-Head' style, despite one of the very few macOS warning messages attempting to persuade the user otherwise. This isn't to say that @ToTo Man did anything untoward, only that, with computers especially, shit happens.
 
This is the first time I've encountered this issue in my almost two decades of using Mac OSX and I don't know what to do...

Every time I boot up one of my recently acquired 2012 Mac Minis, onto which I did a clean install of El Capitan, Spotlight spends the first half hour or so re-indexing my external HDD. I thought nothing of it for the first few days, I even left the machine idle for a whole day so that Spotlight could do its thing, so it's not as if I haven't given it chance to index the drive. But I'm a month in now and it still does it on every startup, even if I haven't added or edited files on the drive.

I read a suggestion to add the affected drive to the privacy tab in Spotlight preferences and then remove it to force a re-index. I tried this but it doesn't seem to have helped. I read on another Apple discussion that replacing the Logic board fixed a similar problem on a MacBook Pro, which seems a drastic course of action!

Is there anything else I can try that's less invasive? I could resort to putting the drive permanently onto Spotlight's privacy list, but then I would presumably be unable to search for files, which would be somewhat inconvenient! I also worry that, if Spotlight is constantly re-indexing my external HDD on every boot, is it doing the same with the internal SSD, and therefore causing unnecessary wear and tear?
I'm not sure what you actually did there. You should add your whole disk to the Spotlight privacy list and then go and have lunch. Then check that the light is still not flashing on the USB drive and maybe still purging the index. Once happy that purging has completed reboot the system and once it comes up remove the disk from the Spotlight privacy list. Spotlight will then re-index all your files and if you open Spotlight you should see the indexing bar. Once indexing has completed the bar will go away.

Are the versions of El Capitan on both the 2010 and 2012 Macs the same? Computers are logical devices and do exactly what they are told so if something works on one machine and not on another and the hardware checks out then the instructions are not the same on both machines.

Have fun,

DV
 
I'm not sure what you actually did there. You should add your whole disk to the Spotlight privacy list and then go and have lunch. Then check that the light is still not flashing on the USB drive and maybe still purging the index. Once happy that purging has completed reboot the system and once it comes up remove the disk from the Spotlight privacy list. Spotlight will then re-index all your files and if you open Spotlight you should see the indexing bar. Once indexing has completed the bar will go away.

Are the versions of El Capitan on both the 2010 and 2012 Macs the same? Computers are logical devices and do exactly what they are told so if something works on one machine and not on another and the hardware checks out then the instructions are not the same on both machines.

Have fun,

DV
The metadata indices are minisclue text files and will be purged almost instantly. You can watch this happen via finder by going to the external volume and then entering ".Spotlight-V100/Store-V2/" via Finder's 'Go', 'Go to folder..." menu choice (alternatively Shift + Command + G) before adding the volume to Spotlight's privacy list (or executing the relative terminal command).
 
Last edited:
Wow, you must be very intellectually agile to be able to make jokes like that. I imagine your wit makes you very popular with your friend. If you have one. Do you get out much?
So back to wonderful Macs...E ****ing G, I have to "eject" USB storage. That's so much better than a PC, where I just airily pull out (hey Beavis, he said pull out). I like that the mac cares enough to give me a little telling off message if I forget and treat it like it's grown up, er a PC, and pull the storage out anyway. Then makes me have to say sorry and perform obeisances if I re connect.
Want to run a third party program on a mac? Better just hope there's a fourth party cludge up available to make it work.
And getting fed up with how speedy and efficient it is? Running too many programs? Just keep on installing those updates.
Okay, different audience. I use a computer as a tool, when I'm not just spaffing about on the net. Call me old fashioned, but I like tools that:
Don't require constant dicking about with and their egos massaging
Do what they're supposed to do, efficiently and ergonomically
Work with other stuff first time, like printers, displays, hard drives
Are repairable and don't have to get chucked away when what would just be a card replacement type failure on a pc happens.
You might like fooling around with computers for its own sake - from what you posted above, I apprehend you do - but for me, well, a means to an end.
Its bad practice not to eject a USB mass storage drive and one day you'll discover that. The usual default for all 3 O/S Linux, Darwin and NT is disk write cache enabled. This improves disk performance as recently accessed stuff is stored in RAM rather than written & read from the disk. This means that the external disk has not been fully updated until the cache has been flushed (written to the disk) and thats what eject does. Its a a game of Russian roulette to unplug a USB mass storage device without ejecting it first.

Cheers,

DV
 


advertisement


Back
Top