In #19343 it appears that read_pos returned "No error" but 0 bytes read.
We need to return an error code in this case, lest whoever called us
think the block was read in correctly after all.
Nothing uses it at present. This saves 8 bytes per PackageNode on 32-bit
and 16 bytes on 64-bit (mostly just EFI). We allocate a PackageNode
for every file in haiku.hpkg, so this is a significant savings in
the small bootloader heap.
This is more efficient and works most of the time. Additionally,
we can potentially join with a previous free chunk in the allocator,
avoiding extra fragmentation on the heap.
app_server (on x86_64) only has 1 "rld heap" area after this change
with a size of 0x50000 (320KB), whereas previously it had around 7
with a total size of 0x80000 (512KB).
The old define was really "check heap integrity", so it's split off.
In the userland test harness, this takes the total time spent in
the allocator from around 1ms to 14ms for all the allocations used
during a boot with "nightly" profile, so it seems acceptable to
enable it under KDEBUG.
Tested with both BIOS and EFI bootloaders, still works.
This uses a recorded array of allocation operations and "replays" them.
The included .h with such an array is mostly for demonstration; the
real captures from bootloader logs are very large (half a MB or more)
and so not included here. A small node.js script that can generate these
from such logs is included.
They were mostly copies of one another, save for the glue code
and a few other things. Now they're mostly unified, and this allows
the test to be greatly simplified, too, since it can avoid including
any bootloader code at all.
The heap implementation itself should have no behavioral changes
from before. Those will come in future commits.
They were hand-rolled singly-linked lists before. This adds 24 bytes
of size to the Team structure, but turns all the removal operations
in parent and process group into O(1) operations instead of O(N) ones.
Realistically, the Thread linked-lists should be converted as well,
but this is trickier due to interdependence on the Team structure.
At present there's only ever one global one, so we don't bother using
an array for multiple screens (and we don't support multiple screens
yet anyway.)
This fixes a very old TODO, and avoids sending a ~32 KB port message
on every application startup.
Note that this breaks the app_server protocol ABI.
The size classes between 4096 and 8192 are very infrequently used
on both 32-bit and 64-bit; a total of 25 objects isn't uncommon
to see across all of them (while 4096 had 68 used objects and 8192
had 49.) We might as well consolidate these and add size classes up to
16384, to take some pressure off the raw allocator.
On x86_64, it seems that we wind up allocating a large number (> 1000) of DMABuffer objects that wind up in the class for 10240, so this
probably saves around 2 MB or so vs. using the raw allocator.
The other new classes have more minor usage (6, 5, and 14 respectively.)
During builds, there are a lot of process arguments (+ environs) that
add up to values between 8K and 16K, so this will benefit that too.
The block size classes seem to not have been changed since their
original introduction in hrev20896 (2007).
Change-Id: Ifff73ed97adf01739fad7f70a1129066925d4b4f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8763
Reviewed-by: waddlesplash <[email protected]>
Every shell execution creates and destroys a FIFO, it seems,
so it's good to avoid the raw allocator (and thus the kernel
VM translation map) here.
After this change, the only remaining use of the raw allocator
during a rebuild of HaikuDepot + the mime_db is the flatArgs
malloc(), and the change to increase the block sizes to 16K
will reduce that to a small number of calls.
* Rename function pointers in struct device for consistency,
and adjust all consumers. Use a switch() in the loop over methods.
* De-indent some code where possible.
* Remove an obsolete comment.
This isn't the BSD behavior, but it saves a lot of time in allocating
large softcs (many larger than the block allocator can handle) as
well as method lookups.
Only one in-tree driver actually seems to try and use the softc
during probe: broadcom570x. We can just add a small patch for it
to skip that set when sc == NULL, as nothing in the method
dereferences it.
Tested with ipro1000, rtl81xx, realtekwifi (USB), all still work (and
of course all other drivers' probe() are called every boot, so
those at least don't have problems when the devices aren't present.)
The IDs were introduced for iflib support, but we can make use of them
in here as well.
Also expose the "resolve_method" function for internal use.
Shouldn't break anything; BSD drivers still seem to work.
This way, we significantly increase the FD table sizes that can
be allocated without needing a "raw" allocation: previously
an FD table size of 512 would've been too large (on x86_64),
while now, tables of up to size 1024 will fit (so long as the
largest block allocator size is 8192, anyway.)
Adjust vfs_resize_fd_table to support allocating tables when
none have been allocated before, and then just use it in
vfs_new_io_context rather than doing the same calculations
and allocations.
No behavioral change intended.
We already didn't inherit FDs, which meant that the only thing we
did meaningfully inherit was the table size. That meant that basically
no applications actually had a table size of the default 256, but all
were at the kernel's 4096 (except Tracker and anything started by it,
as Tracker resets it to 512), and also that basically all applications
had FD tables allocated with the raw allocator instead of the block
allocator, which isn't very efficient.
Since this reduces the default FD table size, some applications
might encounter problems. However, build systems and other such
tools should already increase this by default as needed, and it's
easy enough to patch in calls to setrlimit() if it turns out
some applications needed a higher default after all.
Also remove a redundant call to vfs_exec_io_context. Calling
vfs_new_io_context with the second argument set to "true"
already skips cloning CLOEXEC FDs.
In most cases, we just need a read spinlock and then atomics
when updating this value, significantly reducing lock contention.
But this also paves the way for the use of these methods in
page-related hot paths, e.g. for reserving memory as well as pages
when mapping page tables.
Now that we don't wait the full timeout in most cases, this "optimization"
isn't really necessary; and it was also preventing reserving amounts
of memory that would require both RAM and swap, which is suboptimal
but there's not really much reason to prevent it.
Previously, we would run the low resource manager continously in
a loop until reaching the timeout. This used up a lot of CPU
needlessly, because if the manager fails to release the memory
we need in the first few runs, it's unlikely to do so later.
We could add another waiting mechanism here, but ultimately
it seems to make more sense to just fail early before reaching the
timeout at all, in this case.
Improves system responsiveness under high memory pressure.
vnode_used also acquires it, and acquiring a read-lock recursively
is illegal and can lead to deadlocks. Holding it across a vnode
lock acquisition isn't a good idea either. We only need it to hold it
before acquiring the unused-vnodes lock, and we can actually skip
acquiring that a second time altogether.
Should fix a deadlock observed by PulkoMandy.
If we fail to look up the chain, we won't uninitialize the protocols,
causing a memory leak (or even dangling references.) Add an ASSERT
that would have caught this problem.
Fixes a memory leak when using raw sockets.
Zstd wants a ~90 KB scratch buffer to decompress our 64 KB chunks.
Rather than let it allocate that itself every time, pass in a 2*64KB
"scratch" buffer and statically allocate the working memory from it.
Pass it down using iovecs, and pass down the other buffers in the same
way, to reduce parameters.
Further, rework the object_cache used for heap decompression buffers
to contain objects sized as 4x64KB, so we only need to do one allocation
and deallocation for the compression/decompression and scratch buffers.
Set the minimum reserve to 1 so that the low-memory manager doesn't
reclaim this, as we'll need it when reading back data.
Improves packagefs I/O performance (and thus boot speeds at least a bit,
it appears.)
Change-Id: Id51f6f598b33b9d757a283184c533bb97049529f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8717
Reviewed-by: waddlesplash <[email protected]>
Tested-by: Commit checker robot <[email protected]>
This adds some new parameters to UnmapPage and UnmapPages. The important
one is a "_flags" pointer to UnmapPage, which if specified will be
filled with the page flags instead of PageUnmapped() being called
(and Flush() won't be invoked, either.)
This removes the remaining PAGE_STATE_* changes from VM architecture code.
Change-Id: Iacbc424dd8a75a79986edcd7f04d15a10f773c87
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8728
Tested-by: Commit checker robot <[email protected]>
Reviewed-by: waddlesplash <[email protected]>
All consumers of this API should be adjusted here.
This partially paves the way for use of committed pages for page tables.
Change-Id: Id6fc2edc86fbd80e929c413e23cf8de1509a8215
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8723
Reviewed-by: X512 X512 <[email protected]>
Reviewed-by: waddlesplash <[email protected]>
Tested-by: Commit checker robot <[email protected]>
An extra argument is added to allow the VMAreaMappings objects to
be added to a queue instead of freed directly (and the lock unlocked,
and so on.)
All architectures adjusted.
This means there is now only one place in each TranslationMap that
the page state and other data is directly adjusted (in UnmapArea).
Change-Id: I3ed2d6d969d1b1e235144a1035c90c750779af27
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8716
Tested-by: Commit checker robot <[email protected]>
Reviewed-by: X512 X512 <[email protected]>
Reviewed-by: waddlesplash <[email protected]>
* Add some parentheses for clarity.
* Remove a comment about unreserving pages. In fact this method is
always called with pages already reserved, and in the path the comment
was in, we wouldn't want to unreserve anyway, because we've
successfully allocated.
* Use a boolean rather than an int for "useCached".
No functional change intended.
This cache is used for allocating typically ephemeral KPath buffers
for syscall handlers and the like. Letting it be drained to 0 by
the low resource system will just make future VFS syscalls needlessly
slower, so make sure that doesn't happen.
See inline comment: it's possible to race with other threads
and wind up with pages if there are a lot of waiters or lock
contention. Just bail out and retry in that case.
The only other potential way to fix this I can think of would be to add a
read-write locking strategy; however this would still be prone to races,
since we acquire and release the lock in a loop in reserve_pages,
and keep whatever pages we managed to reserve from the last iteration.
This way, we don't wind up in a situation where there are dozens of
waiters, each needing a handful of pages, but each also having a
handful of pages already reserved, and everything deadlocks.
Significantly improves system responsiveness under high memory usage.
There's still more to be done, however.
It doesn't make sense to reserve memory in other increments.
This would have caught #19295 earlier.
Also use PAGE_ALIGN in VMCache::SetMinimalCommitment.
Nothing uses it, and it isn't correct anyway: vm_page_num_free_pages()
now includes cached pages, too, while vm_available_memory() includes
not just available memory pages but also swap.
The documentation comment claimed this was done but it actually wasn't,
potentially leading to leaks of pages. However, this method appears
to be passed reservation objects only in one rare case in vm_soft_fault
at present. In brief testing, I didn't manage to get that case to
happen.
This method will be used in more cases in future refactors, so it's
important it work correctly.
The DefineBuildProfile logic implicitly removes it later on
when it replaces JAM_TARGETS entirely. But in some cases it
may not do that, and so if we want it to be removed in all cases,
we have to do that here.
Fixes commands like "jam -q -jN @nightly-anyboot haiku.hpkg"
giving a "don't know how to make @nightly-anyboot" message.
e.g. if one merely has "SetConfigVar DEBUG : HAIKU_TOP src system kernel ;"
this won't actually set a value for DEBUG, and thus it won't use one
of the pre-set debug objects directories, and so all the objects will
wind up in the root of wherever jam is run.
If the completion was just notified by another thread, it's possible
we were awoken and are trying to free this CCB before the other thread
released the condition variable's lock. So, call NotifyAll again
to serialize.
Should fix#17522.
* Put variable declarations near use.
* Use AreaCacheLocker, which calls vm_area_get_locked_cache for us.
* De-branchify now that a Locker is being used.
No functional changes intended.
This way, we can avoid printing anything if the superblock magic
doesn't match (i.e. the partition doesn't look like it's of the
filesystem), but print an error if the magic values match but the
rest of the data doesn't.
Reduces syslog spam.
* Remove the "pause" indicator; it adds too much noise at small
sizes and seems confusing at large ones. Instead, draw the
battery in a blue color when it's not in use.
* Reverse the direction of the battery bar. It now has the capacity
on the right and the empty on the left, the opposite of the way
regular progress bars are, but more in line with how this is
displayed on other OSes (and smartphones.)
* Tweak ratios to make the icon display larger in the deskbar.
It now displays closer to the size that regular icon views do.
* Tweak the lightning indicator to be a little larger, also for
legibility at smaller sizes.
* Other misc. cleanups.
Change-Id: I229fe120e769e43e17eeb87638b832418f5e7aaf
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8690
Reviewed-by: Sen Nordstrom <[email protected]>
Reviewed-by: waddlesplash <[email protected]>
List only the packages that are not required by any other package.
Most useful then used as "pkgman search --installed-only --non-required",
but works for non-installed packages too.
Based on OscarL's initial version, rewritten by me to use the real
dependency resolution system instead. One tweak from madmax to get
it to actually work properly.
Co-authored-by: Oscar Lesta <[email protected]>
This reverts a8877df135.
Previously, the "unmergeable" flag was necessary for the RAMFS,
because if the last vnode reference was released while there
was still a consumer (as the old ordering of _RemoveConsumer
had), then the release of the cache reference when the vnode
was removed would result in the cache trying to merge with
its now-only consumer and sole referrer.
Now, instead, we remove the consumer before releasing the store
reference, so that there's no chance the cache will be merged
inside this method.
mmap_cut_tests still pass, web browsers using ramfs shared_memory
still seem to work.
This way, if the Resize() is supposed to take care of the commitment,
it will (and will fail early if it can't), while if we are the ones
responsible for adjusting the commitment, map_backing_store won't
commit at all (avoiding committing far more than will be necessary),
and we can just steal the commitment from the first cache for the second.
This reverts 3a81e9446d (2022).
That commit fixed#17556 by just checking if the area had an
underlying cache that wasn't a RAM cache. But there are cases
where there will be RAM source caches that we have to take
into account, too, not just vnode caches or the like. The
most common example of that would be all areas of a team
after a fork(); the original pages will be in a read-only
source cache.
This commit fixes the real underlying problem: if the first area
has a source cache, then the new second cache needs to have that
as its source, too; and furthermore must have the correct offsets
in order to access its pages correctly.
The test for #17556 that was added in 9ed77019b6
still works as before, as do all the applications I tested that
use cut_area. Some assertion failures that the cut tests triggered
(related to commitment sizes) are fixed by this, as well.
This also seems to fix the remaining instability on fork() in the
boehm-gc's "gctest".
The page_protections aren't changed at all, so all pages that exist
should already have the same protections as are specified in the array.
The only thing different is what cache and area they now belong to, but
the VMTranslationMap does not care about that.
So we don't need to loop over the pages and re-protect them in this case.
We already didn't for all cases where no page_protections were involved.
(It seems this logic was introduced in bdcc293fa8
along with general page_protections support in cut_area.)
When the area has no page_protections but isn't writable,
we also want to use a smaller-than-default commitment.
So, adjust compute_area_page_commitment to handle that case,
and then use it in cut_area where appropriate.
cache->virtual_base is the cache's start address, which no pages will
be found before. area->cache_offset on the other hand is the area's
offset into the cache (i.e. offset 0 in the area will be offset
0 + area->cache_offset in the cache.) These addresses may well be
the same (even if they're not 0), and in many situations they are,
but in situations with shared or cut areas, they may not be.
The only thing that uses this method is madvise(MADV_FREE), which
probably not many things besides the guarded_heap use at present.
We shouldn't return B_OK here, because then the page writer will
assume it's acquired a store ref and can write pages from this
cache, when of course it's done nothing of the sort.
Previously, we'd wind up adding pages from the source to the consumer
that were potentially or actually outside the consumer's bounds.
Now we check the consumer's size and ignore any pages that we don't
want or need; they'll just be freed along with the source cache.
While at it, drop VMAnonymousCache::_MergePagesSmallerSource; it
was the same as the base class's implementation of Merge preceding
this commit; and add a comment to _MergePagesSmallerConsumer noting
that some of the pages may be busy (indeed, I manage to trigger an
assert related to copy-on-write in here at least once.)
I discovered this problem because the page commitment size ASSERT()s
triggered inside Resize() and Rebase(); but the out-of-range pages
already existed in the cache before those functions were called. So,
I've also added an ASSERT to MovePage() that would have caught this
problem more directly.
Augustin Cavalier